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(57) 



ABSTRACT 



A computer interface allows as user to control aspects of its 
behavior by head motion. In one example, a head mounted 
display (HMD) includes a directional detector for generating 
an object detection signal when pointed at certain physical 
objects. The detector is mounted so users can point it by 
moving their heads. In another example, a graphical user 
interface includes the computation of a view space com- 
prised of multiple separate portions, in each of which 
graphical user interface windows generated by separate 
computer processes can be selectively located by a user. The 
user can selectively move such windows between view 
space portions by a point and click interface. The method 
displays a subset of view space portions to the user at one 
time, senses motion of the user's head, and changes the 
portions of the view space displayed, in discrete units of 
view space portions, in response to changes in the position 
of the user's head. In still another example, a graphical user 
interface includes: computing a visual space; displaying a 
subset of space to the user, sensing user motion, including 
monitoring motion of a user's head, so as to divide head 
motions into at least a first and second class; and allowing 
a user to navigate through the visual space in response to 
head motions of said first class, while substantially ignoring 
head motions of the second class for such purposes. 

12 Claims, 34 Drawing Sheets 
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COMPUTER USER INTERFACE WITH virtual reality system was first described by Ivan Sutherland, 

HEAD MOTION INPUT a co-inventor of the parent application of the present 

application, in a seminal article entitled "A head-mounted 
RELATED APPLICATIONS three dimensional display", AFIPS Conference Proceedings, 

Hie present application is a continuation of U.S. patent 5 Vol ^ me 33 ' 1968 This article describes an imaging pipeline, 
application Ser. No. 08/708,103, filed on Aug. 27, 1996, now "lauding: a database for storing all the data, relationships 
U.S. Pat. No. 6,061,064 entitled "System And Method For ^ ^ccts that are relevant to a model to be displayed; a 
Providing And UsingA Computer User Interface With View P 0S1 f 10n f nsor f ° r sclccU f n S a view P 01 " 1 ° f the L model to-be 
Space Having Discrete Portions". Application Ser. No. displayed; a tramformer for traversing the database extract- 
08/708,103 is a File Wrapper Continuation application of 10 m f ths \ a PP ro P™ te data to generate the model from the 
U.S. patent application Ser. No. 08/188,180, filed on Jan. 29, selected view point, and transforming it on the fly to a 
1994, entitled "System And Method For An Improved display J 0 ™* a / rame buffer for stonn S thc transformed 
Computer User Interface" now abandoned. Application Ser. data; and the head mounted display for displaying the data 
No. 08/188.180 was a continuation-in-part of a U.S. patent 5torc !* m L mc framc ^ u ? cr - J** Vlrtual rcallt y s y stcm thus 
application Ser. No. 08/114,546 entided "System And 15 Prides the user with a head-motion paraUax: when the user 
Method For A Computer Video Display" filed on Aug. 31, moves ^. or , hcr hcad L ' the V1CW ^cn through the head 
|o 9 3 mounted display unit changes as it would in real life. 

Another approach to increasing the effective size of the 
BACKGROUND OF THE INVENTION visual space upon which graphic objects such as windows 

j rii ii /. i y . 20 can be placed is to make the view space in which those 

1. Field of the Invention . . # . , , , tl _ r , A t 

objects can be placed larger than the screen and lo place a 

Hie present invention relates to a user interface for map on the screen representing the various parts of that 

computers that allow user head motion to control aspects of expanded desktop. The user can then select a given portions 

its interface. 0 f mat expanded desktop by clicking the mouse on the 

2. Background of the Invention 25 corresponding portion of the desktop map. While this can be 
Computers have vastly increased their ability to process useful, it requires the user to perform mouse operations to 

information. Many computers now include one or more access different portions of the view space, which take time 

powerful microprocessors. Multitasking operating systems and which can be difficult if his computer does not have 

have given the computer the ability to execute more than one mouse or if he, or his bands, are not in situation in which 

application at a time. Application programs have taken 30 using a mouse is convenient. In the mobile computing made 

advantage of this increased computing power, and as a possible by smaller computers and displays, traditional 

result, have become more graphic intensive. computer mice are often difficult and awkward to use, and 

The size of standard computer terminals (e.g., a 19 inch oU ? er met liods of selecting and interacting with computing 

monitor) has become a limiting factor in presenting pro- 35 0D j ects are desirable. 

cessed information to the user. A standard desktop computer __ rww . . „ ™ T „ YVT , 7r ,. WT 

now has the capability to inundate the display terminal of the SUMMARY OF THE INVENTION 

computer with information. The computer industry has it is an object of the invention to provide a user interface 

attempted several approaches to overcome this problem. for computer systems in which a user has provide useful 

The most common approach has been window based ^ input into the interface through use of his or her head; 

software. Windowing systems attempt to maximize the use It is another object of the present invention to make it 

of the screen space of a display terminal by providing easier to access computer information and/or applications. 

overlapping windows and icons. The window operating l4 ■ 4lI 4U * c *u * ■ *- i 

. ^ s , , , . It is still another object of the present invention to make 

environment, although useful, is often frustrating to operate. A „■ , . , c . , 

^ . 7 . ** / « . 11 easier to access windows or groupings of windows in 

The user is required to spend an inordinate amount of time A c „„„ „, *u u- ■ T *- 

. ? r . , , . . . , 45 computing system with a graphic user interlace, 

moving, resizing, and opening and closing various windows 

and icons on the display space. The opening and closing of 11 IS * et an ° lner ob J ect ° f ^ P resent invention to enable 
a window is often slow. Overlapping windows can be users to rapidly navigate through windows or computational 
aggravating to the eye. It is also difficult to manipulate objects. 

information within windows. The physical size of the dis- 50 Jt is still another object of the present invention to make 
play terminal limits the size of each window, the number of it easy to perform certain computer functions without requir- 
windows that can be displayed at a given time, and, in the in S us e of a keyboard or mouse. 

case of graphic intensive applications, is often too small to The present invention relates to a system aad method for 
display an image of an object in its entirety. improving computer user interfaces. 

Another approach to increasing the display surface area of 55 According to one aspect of the invention, a head mounted 
a computer is to simply use a larger monitor. Several display (HMD) apparatus is provided. The HMD apparatus 
companies are marketing twenty-eight (28) inch diagonal includes a display which can be seen by a user wearing the 
monitors. These extra-large monitors do increase the display apparatus. The apparatus also includes a directional detector 
capabilities of the computer to some degree, but the prob- for generating an object detection signal when the detector 
lems outlined above are still present. These monitors are also 6 o receives a certain signal from a physical object in response 
prohibitively expensive to build and difficult to ship to to the detector being pointed at the object. The detector is 
customers. One such monitor currently on the market mounted on the apparatus so a user wearing the apparatus 
weighs over two hundred pounds and is more than thirty can point the detector by moving his or her head, 
inches deep. This monitor is clearly impractical for standard i n some embodiments of this aspect of the invention the 
desktop computers. 65 HMD apparatus includes a see-through window to enable a 

Virtual reality systems represent yet another approach to user to see his or her physical surroundings when wearing 
increasing the display area of a computer. It is believed a the apparatus. In some embodiments the apparatus includes 
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an additional input device to enable a user wearing the FIG. 5 illustrates a video display system pipeline accord - 

apparatus to selectively generate computer readable signals ing to the present invention. 

in addition to those generated by said detector. Id many FIG. 6 illustrates a quantized path-independent view 

embodiment the detector points in the same general d.rec- divided k ^ of ^ £ 

tion as the race of a user wearing the HMD apparatus. In < -,■ * * - \- 

some embodiments the directionfl detector haVa limited 5 aCCOrdmg 10 me present inventl0D ' 

angular view from which it can receive radiation, the user - FIG 7 illustrates a quantized hyper view space compris* 

can point the-angular view by moving his or her head, and in 8 a navigable generalized network of view windows 

the directional detector generates the object detection signal according to the present invention. 

when the detector is pointed at one of said physical object so FIG. 8 illustrates a simplified flow chart of a program for 

that it can receive radiation coming straight from the object. navigating a view space having a combination of the path- 

According to another aspect of the invention, a method of independent and path-dependent features shown in FIGS. 7 

computing a graphical user interface is provided. The and 8, respectively. 

method includes computing a view space in which graphical FIG. 9 illustrates a head mounted sensing and communi- 

user interface windows generated by separate computer y$ ca ting unit, including a visual display, used in certain 

processes can be selectively located by a user. The view embodiments of the present invention. 

space has a plurality rof discrete portion^ each of which is FIG. 10 illustrates another embodiment of a head 

large enough to hold a plurality of windows generated by mounted y ^ & { which cafl be used m ccrtain 

separate computer processes. The method enables a user to embodiments of mc prcsent inven tion. 

selectively locate GUI windows in each of the view space on _ .„ ( 

portions and to selectively move such windows between 2 ° FIG * U e pirates a schematic high level flow chart of a 
view space portions by use of a point and click interface. The P ro S ram ^ altowmg a user to more easily access comma- 
method displays a subset of the view space portions to the ^ va ™us computer applications accord- 
user with a head-mounted visual display, it senses motion of in S t0 the P resent invention. 

the user's head, and it changes the portions of the view space „ FIG - 12 illustrates how the program of FIG. 11 allows a 

displayed to the user, in discrete units of view space plurality of windows to be associated with various locations 

portions, in response to changes sensed in the position of the m a person's office. 

user's head. FIG. 13 illustrates an alternate object sensor for use in the 

In some embodiments of this aspect of the invention, the Dea d mounted sensing and communicating units shown in 

user is allowed to drag a window with a pointing device to 30 FIGS. 9 and 10. 

a desired portion in a view space portion. Id some embodi- FIG. 14 illustrates another alternate object sensor for use 

ments users can change the portion of the view space shown in the head mounted sensing and communicating units 

by rotating their heads. In some embodiments, users can use shown in FIGS. 9 and 10. 

eye movements to input information into the interface. FIG. 15 illustrates the computing apparatus, with input 

According to another aspect of the invention, a method of 35 and output devices, used in certain embodiments of the 

computing a graphical user interface is provided which present invention. 

computes a computer generated visual space. It displays a FIG. 16 is a flow chart illustrating steps performed by a 

subset of this space to the user such as, for example, through con^d interpreter used with a head mounted display of 

a head-mounted visual display. The method senses user tne t e snown m p[Q 

motion including monitoring motion of a user's head. This 40 „ T ^„ , , 

sensing divides head motions into at least a first and second fc . F , IGS ' } 7A X ™ lUus 'f ate a more detailed but still 

class. The method allows a r*er to navigate through the ^ level, version of the flow chart shown m FIG. 11, which 

visual space by changing the subset of the visual space ve u rS1 ° n 15 used in « rta,n embodiments of the invention and 

displayed in response to head motions of said first class, but ^ h ?™T es ^l ^ Sh ° WD 

, i j .. c , , ™. FIGS. 18-34 and 36. 

not in response to head motions of the second class. This 45 

enable a user to move further in the visual space in a given FIG 18 illustrates a screen showing the background 

direction than would otherwise be possible by causing desktop, when no other desktop is shown, with the desktop 

successive head motions in the given direction to be of the manager icon flashing. 

first class, which are counted toward such visual space FIG. 19 illustrates a screen showing the desktop manager 

motion, and causing intervening head motions in the oppo- 50 window with an undefined desktop flashing and awaiting 

site direction — made to prevent the head from being over user definition. 

extended in the given direction — to be of the second class, FIG. 20 illustrates the desktop manager window of FIG. 

so as to go uncounted toward such visual space motion. 19 after the user has defined the previously defined desktop. 

DESCRIPTION OF THE DRAWINGS ^IG. 21 illustrates a screen showing the phone desktop. 

The objects, features and advantages of the system and 55 . FIG * 22 Pirates a screen generated when the trash can 

method of the present invention will be apparent from the 1C0n 15 bcm & dra && ed mto thc trash can deskto P- 

following description in which: FIGS. 23-25 illustrate screens generated when the user 

FIG. 1 illustrates a computer operator using the video selects a window, then points at the trash can, and causes the 

display system of the present invention. 60 Previously selected window to be deleted. 

FIG. 2 illustrates the relationship between a frame buffer, FIG - 26 illustrates a screen generated when the user 

a view port, a view window and a virtual view space of the de-selects a desktop. 

video display system of the present invention. FIGS. 27-28 illustrate screens generated when a window 

FIG. 3 illustrates a block diagram of the video display ^ dragged from one desktop to another. 

system of the present invention. 65 FIG. 29 illustrates a view of portions of a desktop outline. 

FIG. 4 illustrates the video display system generating a ■ FIG. 30 illustrates a view of a portion of a desktop outline 

view window according to the present invention. showing desktops located throughout a user's building. 
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FIG. 31 illustrates a desktop map showing the location 
throughout the user's building of the desktops in the portion 
of the desktop outline shown in FIG. 30. 

FIG. 32 illustrates a view of a portion of a desktop outline 
showing desktops located throughout a user's local geo- 
graphic area. 

FIG. 33 illustrates a desktop map showing locations 
associated with desktops in the user's local area. 

FIG. 34 illustrates a view of a portion of a desktop outline 
showing all the desktops of different users associated with 
one location. 

FIG. 35 illustrates a shared display which can be used by 
the present invention to display different messages for 
different users. 

FIG. 36 illustrates a portion of a desktop outline showing 
desktops intended for display on the shared display shown in 
FIG, 35. 

FIG. 37 illustrates an alternate embodiment of the inven- 
tion using a hand-held computer with a built in display, 
showing how motion of the computer can be used both to 
select desktops and to move the cursor. 

FIG. 38 illustrate an alternate embodiment of the inven- 
tion which uses video recognition to enable the user to select 
computing objects and generate commands by pointing his 
or her hand. 

FIG. 39 illustrates a high level flow chart of a part of the 
program used to recognize and respond to commands gen- 
erated by the user's hand shown in FIG. 38. 

DESCRIPTION OF PREFERRED 
EMBODIMENTS 
L Continuous View Space 

Referring to FIG. 1, a computer operator using the video 
display system of the present invention is shown. The video 
display system 10 includes a swiveling chair 12, a computer 
14 mounted at the base of the chair 12, a platform 16 for 
supporting computer peripheral devices such as a keyboard 
18 and a mouse 20, a head mounted display 22 and a position 
sensor 24 (housed inside computer 14), including a trans- 
mitter 26 mechanically coupled to the head mounted display 
22 and a receiver 28m echanically connected to a stationary 
reference point 30 Jnie"Teference point'can be locatea J abdve : * 
jfBb-useTs fie"Sa* as illustrated in FIG. 1, at the base of the j 
chair 12, or any other stationary location in the vicinit y of] 
the videod^splavsyst&tn 10^^- ■ 
' ReTfefnng to FTg. " 2," tfierelationship between a frame 
buffer, a view port and a view window in a virtual view 
space in the video system 10 is illustrated. The virtual view 
space 40 is the total image area in the video display system 
10. The virtual view space 40 is 360° and has a height of 
135°. The virtual view space 40 is shaped like a "cylinder" 
which surrounds the user. In the preferred embodiment, the 
total size of the virtual view space 40 is equivalent to a wall 
size display having the dimensions of approximately eight 
(8) feet by three (3) feet. 

The virtual view space 40 includes ninety-six (96) million 
discrete points, each identified by a particular yaw and pitch 
location. In the horizontal direction, there are sixteen thou- 
sand (16K) discrete yaw locations. In the vertical direction, 
there are six thousand (6K) discrete locations. 

The frame buffer 42, contained in the memory of com- 
puter 14, includes twelve (12) megabytes (MBs) of dynamic 
random access memory (DRAM) and has a storage capacity 
of ninety-six million memory locations. Each memory loca- 
tion stores pixel information. The memory locations in the 
frame buffer 42 are organized into words. Each word is 
thirty-two (32) bits long. 
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The virtual view space 40 is "mapped" into the frame 
buffer 42. For each (yaw, pitch) location in the virtual view 
space 40, an addressable memory location is provided in the 
frame buffer 42. The memory locations store the pixel 
5 information for the corresponding (yaw, pitch) location in 
the virtual view space 40 respectively. 

The left-most edge of the frame buffer 42 corresponds to 
the 0° location in the virtual view space 40. The right-most 
edge of the frame buffer 42 corresponds to the 360° location 
10 in the virtual view space 40. The bottom edge of the frame 
buffer 42 corresponds to the 0° location of the virtual view 
space 40 and the top edge of the frame buffer corresponds to 
the 135° location of the virtual view space 40 in the vertical 
direction. 

15 During operation, the user may navigate the virtual view 
space 40 by rotating his or her head from side to side, tilting 
his or her head up or down, or swiveling in chair 12. The 
position sensor 24 permits the video display system 10 to 
emulate a video environment that has two degrees of free- 

20 dom. The position sensor 24 generates rotation (yaw) infor- 
mation and vertical movement (pitch) information in 
response to the movement of the transmitter 26 on the head 
mounted display 22 with respect to the receiver 28 at the 
reference point 30. It should be noted that in the preferred 

25 embodiment, only yaw and pitch movement is measured. It 
is within the scope of the present invention to measure other 
motions, such as scaling (forward/backward), roll, lateral, 
side- to -side, and up/down. 

When the user looks at a discrete point in the virtual view 

30 space 40 with the head mounted display 22, the computer 14 
calculates a view point 32 in the virtual view space 40 from 
the yaw and pitch information from the position sensor 24. 
The computer 14 then defines a view port 34 around the 
view point 32 in the virtual view space 40. The view point 

35 32 is located in the center of the view port 34, equidistant 
from the vertical and horizontal boundaries of the view port 
34. The view port 34 has the dimensions of (25 o x20°) within, 
the virtual view space 40. 
To generate an image of the view port 34 in the virtual 

40 view space 40, computer 14 retrieves the pixel information 
from the frame buffer 42 that corresponds to the view port 

| 34. The pixel information is stored in (1120x900) memory 
locations. The pixel information is subsequently transferred 

\ to the head mounted display 22. The pixel information 

45 displayed in the head mounted display 22 is referred to as 
view window 36. The view window 36 includes (1120x900) 
pixels and has the dimensions of (25°x20°) within the virtual 
view space 40. 
The view window 36 may include a horizontal scroll bar 

50 37 which designates the yaw position and a vertical scroll 
bar 38 which designates the pitch position of the view 
window 36. In the example illustrated in FIG. 2, the display 
window 36 is located at a position of approximately 270° 
yaw and 70° pitch in the virtual view space 40. The scroll 

55 bars help the user keep track of the current location in the 
virtual view space 40 and they help to locate regions in the 
virtual view space 40 not currently displayed in the view 
window 36. 

The virtual view space also includes a wrap around 
60 feature. For example, if a view port 34B overlaps the 
(0°/360°) intersection of the virtual view space 40, memory 
locations from both the left side 34B' and right side 34B" of 
the frame buffer 42 are accessed. The pixel information 
contained in regions 34B' and 34B" is used to define the 
65 view port 34B, and are subsequently displayed in the head 
mounted display 22. The virtual view space 40 thus appears 
"seamless" to the user. 
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In summary, the virtual video system 10 of the present 
invention provides an image pipeline. For each yaw and 
pitch sample, the computer 14 determines the view point 32, 
and the viewport 34. The pixel information contained in the 
view port 34 is retrieved and displayed on the head mounted 
display 22 as a view window 36 of the virtual view space 40. 

As the user moves his or her head through the virtual view 
space 40, the frame buffer 42 is immediately traversed. As 
a result, the view window 36 in the head mounted display 22 



match that of the view window 36, and not the size of the 
frame buffer. The window manager is also modified to 
account for the fact that the cursor is primarily controlled by 
the position sensor 24. The window manager insures that the 
cursor always appears in the current view window 36. The 
mouse 20 only secondarily controls the cursor by determin- 
ing its position within the current view window 36. Many of 
these modifications are desirable because the frame buffer 
42 is much larger than the view window 36, which differs 



is continuously updated. The image appears to be fluid and 10 from than prior art video systems where the frame buffer and 

natural. The problems related to lag in prior art virtual reality the display are the same size. 

systems are thus effectively eliminated. In an office environment, for example, the user may be 
II. The Video Display System running a word processing program, a calendar management 
Referring to FIG. 3, a block diagram of the video display program, and a computer automated drawing (CAD) draw- 
system 10 is shown. The video display system 10 includes 15 ing program on the main CPU 50. The information gener- 
a main CPU 50, a system bus (SBus) 52, a graphics ated by these three programs can be displayed, at the user's 
processor 54, a memory 56 including the frame buffer 42 discretion, anywhere in the eight by three foot (8'x3 ( ) virtual 
and code space 57, an arbiter 58, a scan line generator 60, view space 40. For example, when the user opens a window 
graphics output controller 62, first-in-first-out (FIFO) buffer at a particular area in the virtual view space 40, the main 
64, display timing generator 66, the position sensor 24 and 20 CPU 50 is responsible for allocating the corresponding 



the head mounted display 22. 

The graphics processor 54, memory 56, arbiter 58, scan 
line generator 60, graphics output controller 62, FIFO 64 are 
all provided on a single buffer display card 70. The display 
timing generator 66 is an electronic card marketed by 
Reflection Technology along with the head mounted display 
22. A ribbon cable (not shown) is used to couple the display 
buffer card 70 and the display timing generator card 66. The 
two cards are inserted in the housing of computer 14. The 



memory locations in the frame buffer 42. The proper image 
is presented in the head mounted display 22 when the user 
looks at a particular location in the virtual view space 40. 
IV. Video Display Hardware 
25 The graphics processor 54 can be any general purpose 
processor. In one embodiment of the present invention, a 
Fujitsu 40 Mhz SPARQite™ microprocessor is used. The 
graphics processor 54 is responsible for determining the 
view points 32 and view ports 34 in response to the yaw and 



position sensor 24 is wired to the buffer display card 70 30 pitch samples respectively from the position sensor 24. The 



through the RS-232 port (not shown) of the computer 14. 
III. The Computer System and Main CPU 

The computer 14 can be any general purpose computer 
platform, such as a SPARCstation 2 or a SPARCstation 10 
marketed by Sun Microsystems, Inc., of Mountain View, 
Calif., assignee of the present invention. These computers 
run a multi- tasking operating system called Sun OS and use 
a window manager system called Open Windows. 

The primary responsibility of the main CPU 50 with 



graphics processor 54 may also use prediction algorithms to 
predict the movement of the head mounted display 22. These 
algorithms include a smoothing function which reduces the 
effects of instability in the movement of the user's head. A 
35 "stick" function determines how the video display system 10 
is to respond when the user begins to move his or her head. 
Since a user will not keep his or her head perfectly still while 
looking at a view window 36, the stick function will freeze 
the view window. When a movement threshold is exceeded. 



respect to the video display system 10 is to fill the frame 40 the graphics processor 54 "unsticks" the image in the view 

buffer 42 with information to be displayed in the virtual view window 36 and the image can be updated. Such algorithms 

space 40. This information is placed on the SBus 52 by the are well known and are often referred to as "hysterysis" 

main CPU 50, and, under the discretion of the arbiter 58, is algorithms. The graphics processor 54 is also responsible for 

subsequently stored in the frame buffer 42. The manner in calculating and displaying the scroll bars 37 and 38 in the 

which the main CPU places information on the SBus 52 is 45 view window 36. 

well known, and therefore is not described in detail. The memory 56 includes sixteen megabytes (16 MB) of 

The main CPU 50 uses a modified version of Open DRAM. Four megabytes (4 MBs) are dedicated for code 

Windows. The modified version of Open Windows retains space 57 for the graphics processor 54. The code for the 

the ability to move windows, resize windows, open and standard graphics algorithms is stored in this memory space, 

close windows, pull down menus, user interface, etc., within so The remainder of the memory 56 is used for the frame buffer 



the virtual view space 40. Since the virtual view space 40 is 
several orders of magnitude larger than a traditional monitor 
screen, problems related to allocating screen space and 
preventing programs from interfering with one another are 
virtually eliminated. The modified Open Windows, however, 
retains these features in the event they are needed. 

Modifications to Open Windows were made to take 
advantage of and to extend the user interface capabilities of 
the virtual view system 10 environment. These modifica- 



42, as described above. In one embodiment, the memory 
includes thirty-two 4 MBit DRAM chips. 
• The arbiter 58 is an application specific integrated circuit 
(ASIC) chip designed for the video display system 10. The 
55 scan line generator 60, the graphics processor 54, and the 
main CPU 50 all compete for access to the frame buffer 42. 
The arbiter 58 controls access to the frame buffer 42 
according to a basic priority protocol. The scan line genera- 



tor 60, the graphics processor 54, and the main CPU 50 have 

lions predominantly include changes to the window manager 60 first, second and third priority rights to the frame buffer 42 

of Open Windows. For example, when the user opens a new respectively. A state machine is used to select one of the 

window, the window manager ensures that the window is three inputs in accordance with the above-defined protocol, 

displayed in the current view window 36. Similarly, the The arbiter 58 is also responsible for generating the row 

window manager is modified so that dialog boxes appear in address signals (RAS) and column address signals (CAS) for 

the current view window 36. When the user invokes a full 65 accessing the frame buffer 42, for refreshing the DRAM 

screen or a full height function of an image, the window memory of the frame buffer 42, and other memory related 

manager has been modified so that the image is resized to functions. Since bus arbitration and these other memory 
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related functions are well known in the art, a detail descrip- Note that the left side of the view window 36, which is 

tion is not provided herein. superimposed on the frame buffer 42, coincides with the 

The scan line generator 60 is a part of the arbiter 58. A start pixel location. The right side view window 36 passes 

scan line is denned as thirty-six consecutive words (1152 through the last word of the scan line 80. 

bits) in the frame buffer 42. The scan line generator 60, 5 In situations where the wrap around feature is invoked, 

under the direction of the graphics processor 54, is respon- the scan line generator 60, under the direction of the 

sible for addressing the words in the frame buffer 42 which graphics processor 54 will again access thirty-six words, 

define a scan line. Nine hundred (900) scan lines are used to However, a first subset of the thirty-six words are accessed 

generate a single view window 36. from the right side of the frame buffer 42 and the remaining 

The graphics output controller 62 is a shift register 10 words are accessed from the left side of the frame buffer 42. 

capable of holding one scan line and is responsible for The graphics output controller 62 receives the scan line 80 

truncating the scan line to (1120) bits. The truncation from the frame buffer 42 and places its bits into a shift 

eliminates the bits of the scan line that fall outside the register system 82. This system functions as if it were a 1152 

boundaries of the view port 34. The truncated scan line is bit wide shift register, which shifts the 1152 bits so that the 

then segmented into bytes (8 bits). The bytes are stored in 15 start pixel is in the left-most position in the register (i.e., 

the FIFO 64. position 1). Accordingly, the bits in the first 1120 locations 

The display timing generator 66 performs two functions. of the register 82 are used to define a truncated scan line 84. 

First, it provides the bytes stored in the FIFO 64 to the head Any bits remaining in locations 1021 through 1052 arc 

mounted display 22 for display. Second, it generates a new discarded. The truncated scan line is then divided into 8 bit 

window interrupt and scan line interrupt. These two inter- 20 bytes and stored in FIFO 64. 

rupts are used to control the timing of the graphics processor In the preferred embodiment, the size of the actual shift 

54. register is (36) bits wide. For each of the thirty-six words 

The head mounted display 22 provides a (1120x900) that make up a scan line 80, the graphics processor 54 

monochrome pixel image, corresponding to the 1120x900 controls the shift amount by calculating the two's comple- 

pixels of the view window 36 respectively. The view win- 25 ment of the lower five bits of the start pixel address. This 

dow 36 as seen in the mounted display 22 extends (250x calculation determines the offset of the start pixel within the 

200). The head mounted display 22 provides a very clear, scan line 80 and shifts the word in the register accordingly, 

crisp image having a resolution that exceeds a conventional This shift sequence is repeated for the thirty-six words that 

computer monitor. A display device, such as the Private make up the scan line 80. 

Eye™, designed by Reflection Technology, Inc. of Waltham, 30 The truncated scan lines 84 are stored in byte segments in 

Mass., is a suitable choice for the head mount display 22. the FIFO 64 and are sequentially provided to the head 

Although the pixel density of the Private Eye is not quite as mounted display 22 through the display timing generator 66. 

high as that mentioned above, the device could be readily The head mounted display 22 displays the nine hundred scan 

modified to provide the preferred pixel density as mentioned lines, from top to bottom, for each view window 36. In this 

above. 35 manner, the view window 36 is generated and displayed in 

In a preferred embodiment, a "6D Tracking System" by a "raster** like fashion in the head mounted display 22. 

Logitech, Fremont, Calif., is used for the position Sensor 24. In a preferred embodiment, the actual lag time of the 

This device was selected because it is commercially video display system 10 is approximately 0.033 seconds, 

available, is relatively inexpensive, accurate and provides an which is imperceptible to the human eye. To achieve this lag 

adequate yaw and pitch sampling rate. 40 time, the yaw and pitch information is sampled 30 times a 

V. Operation second. The new window interrupt occurs 50 times a second. 

FIG. 4 illustrates the video display system 10 generating and the scan line interrupt occurs 45,000 times a second, 

a view window 36 according to the present invention. When VI, "Video Pipeline 

the graphics processor 54 receives a new window interrupt Referring to FIG. 5, a video pipeline according to the 

from the display timing generator 66, the processor first 45 present invention is shown. The video pipeline 90 illustrates 

determines the location of the view point 32 based on the the functional steps by which the present invention generates 

most recent yaw and pitch sample. Next, the graphics a model to be displayed. This figure and the discussion 

processor 54 determines the view port 34 around the view below highlights the advantages and distinctions of the 

point 32. The view point 32 and the view port 34 are shown video display system 10 over prior art virtual reality sys- 

superimposed on the virtual view space 40. (Note the size of so tems. 

the view port 34 in reference to the virtual view space 40 is The video display pipeline includes a measuring step 92 
not to scale.) The display timing generator 66 then generates for measuring the user's movement, a step 94 for defining 
a scan line interrupt. In response to this interrupt, the graphic the view point of the model to be displayed based on input 
processor 54 determines a "start pixel" location for the first from the measuring step 92, a compulation step 96 for 
scan line 80. The physical memory address for the start pixel 55 modifying the point of view based on a predefined 
location is stored in Dx and Dy registers of the graphics algorithm, and an access step 98 for accessing the appro- 
processor 54 respectively, priate data from the frame buffer 42, and providing the data 

The scan line generator 60 reads the start pixel address to a display device 100, 

information from the Dx and Dy registers of the graphics Hie frame buffer 42 stores all of the possible views of the 

processor 54 and determines the physical address of the 60 model to be displayed. The data, relationships and objects 

word that contains the start pixel in the frame buffer 42. The that are relevant to all possible view points of the model are 

scan line generator 60 then accesses thirty-six consecutive stored in the converted state (i.e., ready for display) in the 

words, in nine bursts of four words, starting with the word frame buffer. This approach significantly differs from the 

that contains the start pixel. The thirty-six words define the prior art virtual reality systems. The need lo store pre- 

scan line 80 which is 1152 bits long. For the scan line 80 65 transformed display information in a database, traversing the 

illustrated in the figure, the first word is designated by bits database, extracting the appropriate data to generate the 

(1-32) and the last word is designated by bits (1120-1152). model from the selected point of view, and transforming it 
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on the fly to the proper display format are all eliminated. As 
a result, the video display system of the present invention 
provides superior performance in displaying those view 
points stored in the frame buffer 42. 

VII. Quantized Path Independent and Hyper View Spaces 5 

Referring to FIG. 6, a virtual view space 110 is shown 
which is divided, or quantized, into an array of screen, or 
view port, sized portions 112. This quantized space is similar 
to that shown in FIG. 2, except that it is shown flat, rather 
than in a cylinder, as in FIG. 2. This space can be created and 10 
viewed- with the system described above by altering the 
computation step 96 shown in FIG. 5 to compute the current 
view port 34 to be the current screen sized portion 112 of the 
view space 110 in which the measurement step 92 deter- 
mines the view point 32 to be located in at any given time. 15 

Such a quantized view space has several advantages. 
First, it insures that the user always sees windows 114 at the 
same position relative to the view port, or screen, in which 
they are viewed. Thus, windows, such as the windows 11 4A 
in FIG. 6, which occupy almost an entire screen will always 20 
be seen fitting within their respective view port. It also 
means that arrangements of multiple windows designed to fit 
within one screen, such as the arrangements 116 shown in 
FIG. 6, will always be seen together within one screen. 
Second, it means the user can much more rapidly search the 25 
view space, because when he or she rapidly traverses the 
space he or she will see a succession of separate screens in 
which the windows are stationary and placed as intended 
within the boundaries of the screen, rather than a blur of 
moving windows which are not properly aligned relative to 30 
the view port most of the time. Third, quantizied view spaces 
allow the user a greater degree of freedom to move his or her 
head without causing the view port to move away from a 
desired view of a given window, because any head move- 
ment within the zone of motion associated with a given 35 
screen causes no screen movement. 

The view spaces shown in FIGS. 2 and 6 are path- 
independent two dimensional spaces. FIG. 7 shows a 
"hyper" view space 120, one which is path dependent. It 
contains a succession of screens, or view port, sized portions 40 
122 that are linked via connections 123 to form a generalized 
network. Each screen-sized portion 122 can have one or 
more such connections 123 to another window, with each 
such connection leading from a given window being asso- 
ciated with a different one of four directions of head move- 45 
ment: up, down, right, or left. The screen the user sees is 
dependent upon the path he or she has taken to get to his or 
her current screen position, rather than being dependent just 
upon current head position or total pitch and yaw movement. 

For example, the portion of the space illustrated in FIG. 50 
7 shows yearly, monthly, weekly, and daily views of a 
calendar program in respectively lower rows 124, 126, 128, 
and 130. In this figure, the connections 123 are represented 
with arrows and the arrowheads indicate the direction of 
those connections. Arrows that are unconnected at one end 55 
connect at that end to screens which are not shown in the 
drawing. In the vertical direction many screens have con- 
nections to other screens which do not have connections 
back to them.. 

Horizontal movement in the year row moves to a view of 60 
the next or last year. In the month row, it moves to the next 
or last month; in the week row, to the next or last week; and, 
in the day row, to the next or last day. Moving upward 
changes from a day, to the week including that day, to the 
month containing that week, and finally to the year contain- 65 
ing that month. Moving down changes from a year, to the 
first month of that year, to the first week of that month, and 
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finally to the first day of that week. Moving up and then right 
will not have the same result as moving right and then up. 
If one starts on the screen for the first week of November and 
moves up and then right, he or she will move to the screen 
for November and then to that for December. If one starts on 
the same screen and move right and then up, he or she will 
move to the screen for the second week of November and 
then to the screen for November. 

Such hyper view spaces allow users to more rapidly 
navigate through a larger number of screens because they 
allow screens to be more densely interconnected than a 
continuous path-independent two dimensional space. But 
such hyper spaces have the disadvantage of making it easy 
for the user to get lost. To alleviate this problem, some 
embodiments of the system have two modes, a 2-D mode in 
which the user operates in a path-independent two dimen- 
sional view space, like that shown in FIG. 6, and a network 
mode in which the user can navigate in a more complexly 
interconnected path-dependent manner. 

In such a mixed space, certain screens 114B perform a 
dual function. They form part of both the path-independent 
and path-dependent parts of the mixed space and operate as 
connections between those parts. For example, assume the 
space shown in FIG. 6 were changed to be such a mixed 
space with such a dual function screen 114B in the upper left 
corner representing a screen of the calendar space shown in 
FIG. 7. In that case, if the user navigated in the path- 
independent space in 2-D mode to reach such a dual- 
function screen and then switched to network mode, he or 
she could then navigate the path-dependent space shown in 
FIG. 7. If he or she then switched back to 2-D mode, he or 
she would be popped back to window 11 4B of the 2-D 
path-independent space. 

FIG. 8 is a high level flow chart of a program loop 132 for 
enabling a user to navigate a mixed type view space in either 
a 2-D mode or a network mode. 

Steps 134,137 and 138 of the loop 132 toggle between the 
2-D and network navigation modes in response to a user 
input. In the embodiment of the invention shown in FIG. 8, 
this input is the detection of the user blinking twice in rapid 
succession. Such a double-blink can be detected by photo 
detectors, such as the detectors 135 shown in FIG. 9, which 
sense when reflection of light off each of the user's eyes 137 
is interrupted. If such a double-blink is detected, then either 
step 137 or 138 is performed. If the system is currently in the 
network navigation mode, step 137 switches to the 2-D 
navigation mode. If the system is currently in the 2-D 
navigation mode, and if the current screen is a dual-function 
screen U4B, step 138 switches to the network navigation 
mode. 

Whether or not the user double-blinks, each pass though 
loop 132 advances to step 140, which tests to see if the user 
has any eye open during the current pass through loop 132. 
If not, step 140 causes the rest of the steps shown in FIG. 8 
to be skipped, causing any head movement while both of the 
user's eyes are closed to be ignored. 

This destroys the advantage of a one-to-one correspon- 
dence between direction of head orientation and screen 
display enabled by the embodiment of the invention 
described above with regard to FIGS. 1-6, an advantage that 
is particularly valuable in many virtual reality applications. 
But such a one-to-one correspondence is impossible with 
path-dependent hyper spaces anyway. In addition, allowing 
head motion to be disregarded when both eyes are closed has 
the same advantage as allowing the motion of a computer 
mouse to be disregarded when the mouse is lifted. That is, 
it allows the user to perform extended navigation in any 
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direction without actually having to move a net correspond- 
ing distance in that direction. Instead the user can intersperse 
motion in the direction counted for navigation with motion 
in the opposite direction which is not counted. This means 
the user does not need to turn his or her head all the way 
around or upside down to perform extended screen naviga- 
tion. It does away with the heed for a swiveling chair 12, and 
it greatly increases the size of a view space which can be 
comfortably navigated by head motion. 

If step 140 finds the user's eyes are open during the 
current pass through the loop, then head position is to be 
used for screen navigation and the remaining steps shown in 
FIG. 8 are performed. If so, step 142 gets the current yaw 
and pitch of the user's head, saves it as the Current Head 
Position, and saves the previous Current Head Position as 
the Last Head Position. 

Next step 144 tests to see if the user's eye was open on the 
last pass through loop 132. It does this by checking a 
variable (not shown) containing this information which is 
saved from the last pass through the loop. If the user's eye 
was closed on the last pass, steps 146-151 are skipped 
because Last Head Position records the head position before 
the user closed his or her eyes, and, thus, the difference in 
head position since then should be disregarded. 

If step 144 finds the user's eyes were open on the last 
loop, step 146 finds the signed yaw and pitch differences 
between the Current Head Position and the Last Head 
Position. Step 147 adds those signed differences to respec- 
tive yaw and pitch sums for all such differences since the last 
time the screen showed to the user has changed. Then step 
148 tests to see if the sum of either the yaw or pitch 
differences is above a respective in-screen threshold for yaw 
or pitch. If not, it causes the user to continue to see the same 
screen. 

If, however, either the sum of yaw or pitch changes is 
above the respective yaw or pitch in-screen threshold, the 
condition of step 149 is met, causing either step 150 or 151 
to show a new screen. If the system is currently in 2-D 
navigation mode, step 150 moves the view port to the screen 
in the 2-D view space which is above or below the current 
screen if the above-threshold sum of pitch changes is 
positive or negative, respectively. It changes to a screen 
which is right or left of the current screen if the above- 
threshold sum of yaw changes is positive or negative, 
respectively. If the system is currently in the network 
navigation mode, step 151 changes the view port to the 
screen in the hyper view space which is connected to the 
current screen by a connection having a direction corre- 
sponding to the direction in which the sum of changes 
exceeds an in-screen pitch or yaw threshold. 
VIII. Head Mounted Displays for Mobile Interactive Com- 
puting 

The head mounted display 22 shown in FIG. 1 substan- 
tially blocks the user's view of the physical world around 
him. This can be desirable when the user is interacting 
almost exclusively with the artificial reality his or her 
computer is creating. But it can be detrimental when the user 
wants to alternate between interacting with the computer and 
the surrounding physical world, or to interact with both at 
once, for example, while walking and talking with other 
people. Also the apparatus shown in FIG. 1 requires that the 
user stay near the receiver 28 to have his or her head position 
accurately read. This limits the user's ability to use the 
invention while moving about his or her office, a building, 
or the world at large. 

FIG. 9 illustrates a head mounted sensing and communi- 
cating unit 161 which enables the user to instantly alternate 
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between viewing its computer display and the external 
world, and which allows the user to navigate a view space 
without needing to stay near the receiver 28. As will be 
described below il also includes features that enable the user 

5 to enter data and select screen objects without the need to 
use a mouse or keyboard, providing him or her with great 
freedom of motion while interacting with the visual display. 

The visual display part of the head mounted unit 161 is 
similar to that currently sold under the name Virtual Vision 

10 Sport by Virtual Vision, Inc. of Redmond, Wash. It is similar 
to the Virtual Vision Sport in that it includes a small back-lit 
liquid crystal display, or LCD, 162, and a lens-mirror 164 
which projects a view of that display into the lower portion 
of a user's field of vision. In preferred embodiments the 

15 display 162 has a resolution higher than that of the Virtual 
Vision Sport, one high enough to be suitable for running a 
graphical user interface such as that provided by Sun Micro- 
systems' Open Windows, described above. 
The head mounted unit 161 also includes a photo detector 

20 135 for each eye. Each such detector determines when its 
associated eye is opened or closed by detecting the amount 
of light reflected off the ball of that eye. The unit 161 also 
includes a microphone assembly 166 similar to that cur- 
rently found on many telephone head sets to enable the user 

25 to enter speech information, and one or more ear phones 168 
to enable the user to hear audio output. 

The head mounted unit also includes a bar code reader 
170 of a type currently available from Symbol Technologies 
of Bohemia, N.Y. This bar code reader is capable of scan- 

30 ning a laser beam in a rectangular raster pattern of scan lines, 
and of reading bar code patterns which fall within thai 
rectangular scan pattern at a distance of twenty feet or more. 
It does so by detecting how the intensity of light reflected 
from the laser beam varies in correspondence with the 

35 progression of that beam along the scan pattern. This bar 
code reader is for use in location quantized view spaces in 
which the user navigates by motion relative to locations 
external to the visual display. Such view spaces are 
described below with regard to FIG. 12 and many of the 

40 figures that follow it. For applications like those described 
above with regard to FIGS. 1-8 such a bar code reader is not 
necessary. 

The head mounted unit 161 also includes other electronic 
component necessary to interface to or drive the display 162, 

45 the photo detectors 135, the microphone 166, the speaker 
168, and the bar code reader 170, as will be understood by 
those skilled in the art of electronic design. It also includes 
wires 172 which communicate signals from the head 
mounted unit to a computer. Preferably this computer is 

50 small enough to be worn by the user, such as in a back pack, 
so the user is free to walk around while using the computer. 
Such a computer can be constructed from a SPARCstation 
computer, of the type described above, with the standard AC 
power supply replaced with a battery power supply. A 

55 computer with enough power to run Sun OS and the Open 
Windows graphical user interface could be built to fit within 
a user's pocket using today's technology. Within the coming 
decade such computers are expected to be small enough to 
fit within a head mounted unit such as that shown in FIG. 9. 

60 The visual display in the head mounted unit shown in 
FIG. 9 is not a see -through display, since the mirror 164 
through which the user sees the visual display is not trans- 
parent. In addition its display is located in the lower part of 
the users field of vision, at a location which some might find 

65 uncomfortable to look at for extended periods of time. 

FIG. 10 provides a schematic illustration of an alternate 
embodiment 161A of a head mounted unit which is similar 
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to that shown in FIG. 9, except it includes a true see-through graphical user interface the communication links are usually 

visual display. In this embodiment a linear array of light windows created when an application is opened. Such 

emitting diodes 176 form the individual pixels of each video windows communicate from their application to a user by 

scan line. Its light pass through a lens 177, then bounces off providing visual information or other output. They commu- 

mirrors 178 and 180 onto a rapidly vibrating, partially 5 nicate from the user to their application in response to user 

transparent mirror or dichroic lens 182, from which it is then mouse clicks, mouse drags, or typing on objects shown in 

reflected into the users field of vision. The mirror 182 the window. In character based computers the communica- 

vibrates about an axis 183 in synchronism with the -vertical tion links can be screens or non-graphic windows. In audio 

scan period of the video image, spacing the reflection of computers the communication links can be connections to 
each horizontal scan line in the user's field of vision, so as 10 audio input and output. 

to form a complete screen image. This is similar to the Step 155 performs computation on a user interface when 
Reflection Technology display 22 described above with user input or the computation in steps 153 or 154 require it. 
regard to FIG. 1, except that the mirror 182, and any material The computation of the user interface includes, among other 
on the opposite side of it from the user, are made of largely functions, receiving user input, selecting which communi- 
transparent material. Preferably a separate visual display 15 cation links to direct such input to, and enabling the user to 
apparatus is provided for each eye, and the bar code reader choose a selected communication link for association with a 
170 shown in FIG. 9 is placed between them. location which is external to the computing apparatus run- 
Further information on miniature displays from Reflec- ning the program of FIG. 11. 
tion Technology, Inc. is provided in U.S. Pat. No. 4,902,083, Step 156 responds when the user has been detected 
entitled "Low Vibration Resonant Scanning Unit For Min- 20 entering input selecting a given communication link for 
iature Optical Display Apparatus" issued to Benjamin Wells association with a given external location by causing step 
on Feb. 20, th 1990; U.S. Pat. No. 4,934,773, entitled 157 to be performed. Step 157 makes the association by 
"Miniature Video Display System" issued to Allen Becker storing it in memory. 

on Jun. 19, 1990; U.S. Pat. No. 5,003,300 entitled "Head Step 158 detects when relative motion is directed toward 

Mounted Display For Miniature Video Display system" 25 the external location. Such relative motion includes any 

issued to Benjamin Wells on Mar. 26, 1991; U.S. Pat. No. motion directed between the user or a sensor which is part 

5,009,473, entitled "Low Vibration Resonant Scanning Unit of the computer and the external location. For example, it 

For Miniature Optical Display Apparatus" issued to Gregory includes the user's pointing at the location; his or her 

Hunter et al. on Apr. 23, 1991; and U.S. Pat. No. 5,023,905 moving toward, or into, the location; or the location's 

entitled "Pocket Data Receiver With Full Page Visual Dis- 30 coming within a given distance of the user, or his or her 

play" issued to Benjamin Wells et al. on Jun. 11, 1991, all computer, or within a direction in which he or she, or a given 

of which are incorporated herein by reference. sensor, is pointing. 

A see-through display, such as the one shown in FIG. 10, When step 158 detects such relative motion toward an 

has an advantage of enabling a user to watch video infor- external location associated with a given communication 

mation in the center of his or her field of vision where it is 35 link, it causes step 159 to select that given communication 

easy to watch. It also makes it easier for the user to monitor link. Such selection can include causing the link to be placed 

both a computer image and information from the surround- on the display screen if it is a visual object, causing input or 

ing world at the same time, since only a change in focus is output to be directed through that communication link, 

required to switch between the two. Such a see-through performing a function associated with that link's 

display has the disadvantage, relative to the display of FIG. 40 application, or performing a function upon that link or its 

9, of placing the image on a transparent surface where it can associated application. 

interfere with the user's perception of the world around him, The loop 152 also performs other functions necessary in 

or in which light from the surrounding world can interfere the operation of its computer, as is indicated by the ellipsis 

with such images. in FIG. U, before making the next pass through the steps 

IX. Selecting Application links by Motion Relative to 45 described above. 

External Locations X. Location Quantized View Space 

FIG. 11 is a simplified high level flow chart representing FIG. 12 illustrates the operation of a preferred embodi- 

an aspect of the present invention. According to this aspect, ment of the aspect of the invention described above with 

operation of a computer is controlled by motion relative to regard to FIG. U. In this embodiment the user wears a head 

locations external to that computer, such as by the user 50 mounted unit, like the unit 161 A described with regard to 

pointing at a selected object in his or her office or entering FIGS. 9 and 10, which includes a head mounted display 

a given location in his or her building or town. which allows the user to see the world around him or her and 

The program loop 152 shown in FIG. 11 repeatedly which includes a bar code reader 170. In this embodiment, 

performs a sequence of steps 153-159. Step 153 performs the communication links are windows 180 in a graphical 

computation on each of a plurality of applications which 55 user interface. The view space of this user interface is 

currently requires computation. An application requires divided into a plurality of separate parts 182, called 

computation, for example, if it has received user input desktops, each of which is associated with an external 

requiring a response during the current pass through the loop location, in this case the location of a bar code pattern 184. 

152, or if it has yet to complete a task started in a prior pass The user associates a window 180 with an external location 

through that loop. In a personal computer such applications 60 by placing it in the desktop associated with that location. The 

can include, among others, word processor, spread sheet, system detects relative motion toward an external location 

calendar, phone book, electronic mail, data base, and custom with which a bar code pattern has been associated when the 

applications. In an audio computer, for example, such appli- user points his or her head at it. When this happens the bar 

cations might include voice mail, dictation, and telephony. code pattern will fall within the radial zone 185 scanned by 

Step 154 performs computation on a communication link 65 the bar code reader 170 which extends out from the front of 

associated with an application when user input or compu- the head mounted unit. This causes the bar code reader to 

tation in steps 153 or 155 requires it. In computers with a read the pattern 184, and the system responds by displaying 
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the desktop 182 associated with that pattern in the visual circuit 208 for decoding temporal modulations in the infra- 
dilaTof the user's head mounted" unit. This display ^^^^^^« 2 ^^;^^^ 
includes any windows 180 previously located in that of the invention the user places a smdl mtrared emiue runu 

code 184A near the calendar 186, and has associated a ticular pa[tern> and a battery for powering the 

desktop 182A with that bar code which includes a monUi- Jj^de ^ coding circuk ^ head mourjtcd m i t 161C 
view window 180A from a calendar application. Similarly blocks all light from reaching the infrared receiver except 
the user has placed a day view window 180B from the same 10 mat which p&sses through a small hole 209 located directly 
application in the desktop 182B associated with the bar code in front of tnat receiver. As a result, the receiver will only 
pattern 184B located near the wall clock 188. The user has () etect a transmitter's light if that transmitter is in a limited 
placed a window 180C from a phone book application in the angular zone 211 which extends radially out in the direction 
desktop 182C associated with the bar code pattern 184C on m which the head mounted unit is pointed, 
the phone 190. 15 FIG. 14 shows another embodiment 161D of the head 

It can be see that, in this example, the user has attempted mounted unit which contains a video camera 212, instead of 
to place bar code patterns near physical objects which bring a bar code reader or an infrared detector. In this embodiment 
to mind the particular windows found in the desktops the computer 14Aused with the head mounted unit 161D 
associated with those objects. Thus, a print manager window contains a visual processing unit 214 w^ch supplies visual 
is found in the desktop associated with the printer 196, 20 information to a visual recognition module 216^ This module 
windows relating to the user's personal life in a desktop compares signals received from the camera 212 with pat- 
associated with a family picture 198, electronic mail in a terns stored in a visual pattern memory 218. These patterns 
desktop associated with an in/out basket 200, and a window represent signals previously received by the camera 212 
associated with blue sky ideas in a desktop associated with when pointing at particular objects, such as the phone 190. 
a window 202 Thus, the physical objects on, or near, which 25 When the recognition module 216 determines that s.gnals 
the user places bar code patterns can function as physical received by the camera 212 appear to match the signal 
icons. That is, they can function to remind the user which pattern representing a given object, it supplies an output . 230 
windows are in the desktops associated with which bar indicating that the object has been recogn.zed. If the user has 
codes, and they can act as a visual reference which tells the previously associated a desktop with the objec 
user where to point so the bar code reader 170 will read the 30 the system will then cause the desktop to be shown m the 
bar code that brings a desired desktop or window into view. user's visual display. 

Preferably the bar code reader 170 is one of a commer- XI. Hardware for Ucation Quantized View Spaces 
cially available type which uses an infrared laser beam, and FIG. IS is schemat c representation of computing appa- 
me bar code patterns 184 are written in infrared ink on clear ratus 230 which enables a user to associate desktops with 
upe Thisreducesthevisualimpactofthebarcodepatterns, 35 locations, select desktops by motion re laUve tc .such 
so they do not significantly impair the appearance of the location, and to manipulate visual objects in hosedeskt ops. 
obiects on or neaf which they are placed. This apparatus preferably includes a portable general put- 

Tnl wmdow 180D associated with the bar code pattern pose computer 14A and a plurality of input and output 
184D on the trash can 192, and the window 180E associated devices. 

with I pattern 184E neaV the picture of the hide-desktop 40 The general purpose computer 14A is preferably small 
icon 194 are unusual in that the applications with which they enough to be easily carried or worn as described abo ve with 

re connected are operator functions. That is, they are regard to the head mounted unit 161. It can be consmicted 
tactions which perform operations on other desktops, for example, from a SPARCstation computer of the type 
:r n rws,ormeappUcationsa P ssociatedwi.hthosevvmdows. described above, with the standard AC power supply 
As will be explained below in greater detail, the trash can 45 replaced by a battery power supply. The computer 14A 
wtndow is used to delete other objects which have been preferably runs Sun OS and Open Windows, with he 
seated by the user and the hide desktop icon is used to modifications to that software necessary to incorporate the 
remove the last selected desktop from the visual display. features shown in FIGS. 16 and "A-17D. 
™ bar code 184F is unusual in that it is worn as part of In an embodiment using the ^ead mounted unit ^161A the 
an identity badge on the clothing of a person 204, rather than 50 computing apparatus 230 mcludes the visual display 162 
being stuck on an inanimate object. The windows 180F microphone 166, and speaker, or earphone, 168 of that head 
Seed in ia abated desktop 182F are those which a user mounted unit, and all the standard interface hardware and 
Ses to usHhen he or she sees the particular person software necessary to interface to such mput/output devices. 
reSgtheTarcodepa«.ernl84F.Onesuchwindowmight The apparatus 230 preferably -includes connections for a 
mdude information about the person wearing the bar code, 55 keyboard 232 and a mouse 234. But, as »s explained I below 

u c h as aa a out of a corporate'personnel directory. Another such input dev ces are often not necessary in p referred 
mi include matters which the user wishes to discussed embodiments of the invention. The apparatus includes the 
wth he person, and so on. Other desktops, associated with photo detectors 135 for detecting when each 
b codes worn by other persons, can contain windows are opened ordosedJt also mcludesa compassed ector236 
associated with such other persons. » located on the head mounted urn 161A ^ Thus detector 

In alternate embodiment ci the invention other types of includes a magnetic sensor which detects changes in y aw of 
detectors besides bar code readers can be used to detect the head mounted unit relative to the earth s magnetic feU 
when the user's head is pointing toward an external location lines. The apparatus includes a gravity driven tilt indie to 
" t v 238 to detect changes in pitch of the head mounted unit. It 

01 HG13 shows a head mounted unit 161C which is similar 65 further includes the bar code detector 170 for detecting when 

to the units 161 and 161A, except that instead of a bar code the head mounted unit is po.nted at a location labeled w.th 

reader it includes an infrared receiver 206 and a decoding a given bar code pattern. 
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The apparatus also includes an intra -building location If the eye detectors 135 find both of the user's eyes are 

detector, or locator, 240. This detects the computing appa- open, it moves the cursor in the visual display in correspon- 

ratus's location relative to the interior of buildings. It dence to head motion made while both such eyes are open, 

includes a global positioning system, or GPS, locator 242 That is, the cursor's X position is moved in proportion to 

which detects the apparatus's location on the surface of the 5 changes in the head's yaw indicated by the compass detector 

earth. It further includes a high frequency wireless LAN 236, and the cursor's Y position is changed in proportion to 

transceiver 244, which enables it to be connected to a changes in head pitch indicated by the tilt indicator 238. 

network of other computers, including other mobile com- If the eye detectors find that the left eye isxlosed, but the 

puters like the apparatus 230 of FIG. 15, itself. right eye is opened during such head motion, step 262 moves 

The intra-building and GPS locators are provided to 10 the cursor in a similar manner as step 261, except that it 

enable the user to associate information with building or performs a left mouse button drag if the user moves his or 

geographic locations and to select that information when he her head. If the eye detectors find the right eye is closed and 

or she enters those locations. GPS locators are well known the left eye is open during such motion, step 264, performs 

and commercially available. They can determine one's loca- a right mouse button drag. And if the eye detectors find the 
tion on the surface of the earth to with approximately 3 feet, 15 user rapidly blinks in one eye followed quickly by keeping 

based on signals received from satellites. Unfortunately they the other eye closed while moving the head, step 266 

often do not work well indoors. For that reason a separate performs a middle mouse button drag, 

intra-building locator is provided. If the eye detectors 135 detect a blink of the left eye by 

The wireless LAN enables the user to be connected to a itself, step 268 generates the equivalent of a left mouse 
network while having freedom of motion, and, as is 20 button click. If they detect a double-blink of the left eye by 

described below, it enables users to associate information itself, that is two rapid blinks in quick succession, step 270 

with specific locations on other user's systems. generates the equivalent of a left mouse button double-click. 

XII. Command Interpreter for Head Mounted Display Used If they detect a blink of the right eye by itself, step 272 

with Location Quantized View Space generates a right mouse button click. If they detect a 

FIG. 16 is a high level flow chart of a command inter- 25 double-blink of the right eye by itself, step 274 generates a 

preter which allows a user to control many of the functions right mouse button double-click. If they detect a rapid blink 

of the computing apparatus 230 with a head mounted unit, in one eye followed quickly by a blink in the other, step 276 

such as 161A described above with regard to FIGS. 9 and 10. generates a middle mouse button click. And if they detect a 

The command interpreter is part of the loop 250 of a larger rapid blink in one eye followed quickly by a double-blink in 
command interpreter which dictates the response of the 30 the other, step 278 generates a middle mouse button double - 

apparatus 230 to user inputs. The particular command inter- click. 

preter shown in FIG. 16 is optimized for a see-through Thus, it can be seen that the command interpreter shown 

display, such as the one shown in FIG. 10, which has in FIG. 16 enables a user of a see-through head mounted 

separate displays and photo detectors for each eye, and in display to turn on and off the display in an instant, to select 

which the visual display is project in front of the user's view 35 windows by pointing at, or coming within, locations asso- 

of the surrounding world. ciated with desktops, and to move and click the mouse 

Step 252 of the command interpreter responds to the cursor, all without the use of his or her hands, by head and 

detection of a double-blink by the eye photo detectors 135 eye movements. If the computing apparatus is supplied with 

shown in FIGS. 9 and 15. It responds to such a double-blink speech recognition software, and interpreter makes it pos- 

by toggling the visual display between on and off modes. 40 sible to use a powerful computer at any time and place, as 

That is done so the user can quickly and easily turn on and a user moves about, without the need to hold or touch 

off the see-through display whenever desired, to minimize anything. 

its interference with his or her perception of the outside XII. Flow Chart and Screen Displays for Location Quari- 

world. tized View Space 

Step 254 responds to an indication by one of the external 45 FIGS. 17A through 17D provide is a simplified flow chart 

location indicators, such as the bar code reader 170, the of that part of the user interface of the computing apparatus 

intra-building locator 240, or the GPS locator 242, that the 230 which relates to the association of desktops with exter- 

user is pointing, at or has come within, a given location. If nal locations. FIGS. 18-34 represent part of the graphical 

so, either step 256 or 258 is performed. output produced by the program represented by that flow 

If the visual display is currently in its off mode, step 25 6 50 chart, 

shows a labeled icon of the desktop associated with the Like the steps of FIG. 16, the steps of FIGS. 17A-17D are 

detected location at the bottom of the screen, to alert the user part of the command interpreter loop 250 of the computer 

that there is a desktop for him or her to see without 14A. The first step of this part of the command interpreter, 

substantially interfering with his or her vision of the outside step 254A determines whether or not a location associated 

world. If the display is currently in the on mode, step 25 8 55 with an ID has been detected, such as by the bar code reader 

displays the current desktop and any windows within it, 170, the intra-building locator 240, or the GPS locator 242. 

giving the last selected window within that desktop the The step 254A shown in FIG. 17A is another description of 

current focus. the step 254 shown in FIG. 16, except that it assumes for 

Step 260 uses the eye detectors 135 to determine if any of simplicity that the visual display is on and it focuses more 
the user's eyes is open and has been open since the last pass 60 on the responses made to different kinds of IDs and desk- 
through the command interpreter's loop. If so, it causes steps tops. 

^cc — a u„ n A .,,;«u a: — t*„* — iw^,. — ♦„ — m ;* „;*u mt 
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cursor motion. Step 260 performs the same function as steps 
140 and 144 of FIG. 8. This is, it allows the user to 
selectively cause certain head motion to be ignored, so he or 65 
she can perform extended navigation in any direction with- 
out requiring corresponding net movement in that direction. 



to be performed, depending upon whether or not a desktop 
is currently associated with the detected ID for the user. 

If no desktop is associated with that ID for the user, step 
302 pops up the desktop manager icon 306 at the bottom of 
the display screen 308, as shown in FIG. 18. This is done to 
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enable the user to associate the ID of a new bar code pattern 
with a desktop. This is necessary for bar code patterns, 
because they are external objects about which the computer 
14A initially has no knowledge other than the IDs defined by 
their pattern. 

If the user double-clicks on the desktop manager icon 
presented by step 302, step 310 causes the steps 311 and 328 
to be performed. Step 311 pops up the desktop manager 
window 312, as shown in FIG. 19, with a potential new 
desktop entry 314 blinking at the top of the user's desktop 
outline 316. 

The desktop manager 312 allows a user to perform many 
of the functions involved in associating desktops with 
locations, windows with desktops, and properties with desk- 
tops and windows. The desktop manager contains a window 
showing a portion of a desktop outline 316. A larger view of 
this outline is shown in FIG. 29, with ellipsis indicating 
portions of the oudine which are not shown in that illustra- 
tion. This outline is a hierarchical representation of which 
desktops contain which windows for which users, and which 
desktops are associated with which locations. 

In this outline a user is indicated by a display icon 318 
followed by a user's name, such as the name 320 shown in 
FIG. 19. The display icon 318 represents the user's visual 
display in which the desktops under that user's entry will be 
seen. A desktop is indicated by a desktop icon 322 followed 
by a desktop title, such as the name 324. Desktops can be 
placed under other desktops in the outline, just as the 
desktops 322 A have been placed under the desktop 322 
labeled "MY OFFICE AT WORK". As is shown in FIG. 29, 
windows are indicated by window icon 323, followed by an 
application name and file title. 

Desktop icons representing desktops associated with 
external locations are preceded by location icons, which 
indicate the type of location each such desktop is associated 
with. Bar code location icons 326 A indicate that their 
associated desktops are associated with bar code pattern. 
Intra-building location icons 326B, shown in FIG. 30, indi- 
cate their desktops are associated with locations defined by 
the intra-building location system. GPS location icons 326C, 
shown in FIG. 32, indicate their desktops are associated with 
locations defined by the GPS location system. 

In the example shown in FIG. 19, the potential new 
desktop 314 has a bar code location icon 326 A, since it 
popped up because a new bar code pattern was detected. If 
the user selects to move or enter a name for the potential 
desktop associated with the bar code pattern, step 328 will 
create an empty desktop in association with the ID, so the 
next time that desktop's ID is detected step 304 will be 
performed instead of step 302. FIG. 20 illustrates the desk- 
top outline after the user has moved and named the potential 
desktop 314 to have the location and appearance shown. If 
the user does not name or move the potential desktop 314 
shown in FIG. 19, the bar code ID which gave rise to it 
remains unassociated with a desktop for the user, and the 
next time that ID is detected step 302 will be repeated. 

If a detected ID has a desktop associated with it in a user's 
portion of the desktop outline, step 304 will cause either the 
steps under step 330 or 332 to be performed, depending on 
whether the ID's associated desktop has had an operator 
function substituted for it. As is explained below, if an 
operator function has been substituted for a desktop, when 
the desktop is selected its substituted operator function is 
performed instead of the desktop being displayed. 

If the desktop associated with the detected ID does not 
have an operator function substituted for it, step 330 causes 
steps 334-340 to be performed. 
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Step 334 removes the previously selected desktop from 
the display if its has not been designated as a persistent 
display. As is explained below, the user can designate a 
given desktop to be non-persistent, semi-persistent, or per- 
5 sistent. Non-persistent desktops remain on the display only 
when their corresponding ID is being detected. Semi- 
persistent desktops remain on the display from the time their 
associated ID is detected until the ID for another desktop is 
detected. Persistent desktops remain on the desktop 
10 (although they may be covered by another desktop) until the 
user explicitly selects to close them. 

Step 336 sounds the audio prompt, if any, associated with 
the desktop or any windows within it. Such audio prompts 
can contain a tone to draw the user's attention to a desktop 
15 if the display is in the off mode described above with regard 
to FIG. 16, or it can contain an actual audio message. 

Step 338 makes the desktop assuaged with the detected 
ID the selected desktop. It displays that desktop at its last 
displayed location and size, covering any other desktops in 
20 that portion of the display, showing any windows of the 
portion of the desktop displayed, and giving input/output 
focus to the window last selected in that display. 

For example, if the user points his or her head toward the 
phone 190 shown in FIG. 12, the bar code reader 170 will 
25 detect the bar code pattern 184C and step 338 will display 
the desktop 182C shown in FIG. 21, and its included 
windows 180C. It will make the window labeled Phone Mail 
the selected window for input and output, since, in the 
example, it was the last selected window in that desktop, A 
30 desktop is a graphic window. Like other windows it can be 
dragged, resized, closed, and reduced to an icon and then 
re -expanded. But it is unlike most other windows, in that 
windows associated with one or more different applications 
can be located within it. 
35 If the user is performing a drag on a window with the 
mouse cursor 343 when step 338 displays a new desktop, 
step 340 will cause the drag to continue with the dragged 
window being placed over the desktop displayed by step 
338. If the drag places the mouse cursor inside the newly 
40 displayed desktop, step 342 will move or copy the window 
to the window's current location relative to the desktop, 
depending upon whether the drag is a move or copy drag. 

Assume for example the user places the cursor 343 on the 
window formed by the trash can icon 180D shown in FIG. 
45 21, and copy drags that window up to the position 341 
shown with dotted lines in that figure. If, without letting go 
of the drag, he or she then points his or her head at the trash 
can 192 shown in FIG. 12, the trash can desktop 180 would 
appear under the desktop, as shown in FIG. 22, assuming the 
50 trash can window, or icon, 180D has not yet been substituted 
for that desktop. If the window 180D were a normal window 
it would be moved to the new desktop 180 of that figure as 
soon as the user let go of the drag. 

But if the window being dragged is the icon of an operator 
55 function, such as the trash can icon 180D, step 344 pops up 
a dialog box (not shown in the figures) asking the user if he 
or she wishes to substitute the operator function for the 
desktop, meaning future selection of the desktop will cause 
the icon's associated operator function to be performed. The 
60 dialog box also allows the user to specify that a warning be 
given before the operator function is performed in response 
to the detection of an associated ID. If the user selects not 
to substitute the operator function for the desktop, the icon 
is merely placed in the desktop like any other window. 
65 If the desktop associated with an ID detected by step 254A 
has an operator function associated with it, step 332 causes 
either steps 346 or 350 to be performed. If the user has 
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selected that the operator function have a warning before 
being evoked by detection of the desktop's ID, step 346 
causes step 347 to repeatedly flash both the selected object 
upon which the operator will function, if any, and the icon 
for the operator function. If the user continues to point at the 
location associated with the operator function for four 
flashes, step 348 performs the operator function on the 
selected object. If the operator function does not require a 
warning, step 350 immediately performs the operator func- 
tion on the object, if any, which has previously been selected 
for it. 

For example the trash can icon 180D deletes any graphic 
object which is dragged onto it. The function performed by 
the trash can icon can have significant consequences. Thus, 
users will normally require a warning before its function is 
performed when they point at the physical trash can 192, 
since they might point at that physical object without 
thinking about its associated function. Thus, if the user has 
the mouse cursor 343 on a draggable object, such as the 
window 180G in FIG. 23, with the mouse button down so as 
to perform a drag when the user points his or her head at the 
trash can 192 in FIG. 12, that draggable object and the trash 
can icon 180D will flash, as is indicated in FIG. 24. If the 
user does not move his or her or her head, or let go of the 
mouse button by the end of the fourth flash, the trash can's 
application will delete the draggable object, causing the 
screen to appear as in FIG. 25. 

The close-desktop icon 180E, shown in FIG. 12, closes 
the currently selected desktop, if it is selected by looking at 
the bar code 184E placed next to a picture 194 of that icon 
shown in FIG. 12. This allows a user to quickly remove a 
desktop which has popped up without needing to first select 
it with the mouse cursor, as would be necessary to close it 
with the trash can icon. 

If the ID associated with a selected desktop is no longer 
detected, and if the selected desktop is non-persistent, step 
354 de-selects that desktop and removes its from the display. 
As is explained below, the desktop manager allows a user to 
selectively make desktops either persistent, semi-persistent, 
or non-persistent. If the currently selected desktop is 
persistent, it will not be removed from the screen (although 
it might be covered by another object) unless the user 
specifically selects to, such as with the close desktop button 
343, shown in FIG. 21, or the hide-desktop icon 108E. If the 
desktop is semi-persistent it can be closed by the explicit 
steps which close persistent desktops, and, in addition, it will 
be closed by step 334 every time another desktop's ID is 
detected by step 254A. If it is non-persistent it will be closed 
in all the situations in which semi-persistent desktops are 
closed, and in addition, by step 354 if its ID is no longer 
being detected. 

When all desktops have been removed from the visual 
display the user sees the background desktop 358, shown in 
FIG. 26, which normally is clear, except where the user had 
placed windows or icons upon it. This desktop has a special 
desktop icon 360 in the desktop outline, shown in FIG. 19, 
representing a larger desk, since it is the main desktop, the 
one the user sees when no other desktops are shown. 

Steps 362, 364, 366, 368, 370, 372, 373, and 380 list some 
of the steps that will cause a particular type of action to be 
performed in response to a particular type of user input from 
a pointing device, such as a computer mouse or input 
simulating a mouse in response to head and eye movements 
as described in FIG. 16. 

If the user drags a window by grabbing its title bar with 
the mouse cursor, step 362 moves an outline of the window 
during the drag, and places the window at its new location 
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when the drag ends, making it the selected window in the 
desktop it is in. During such drags the window's outline will 
not extend outside the desktop in which the mouse cursor 
which is dragging it is located. If the cursor crosses from one 

5 desktop into another, a tone sounds, and only that part of the 
window outline which is in the new desktop the cursor is in 
is shown. If the user ends a drag with the window in a 
different desktop than that in which he or she started, step 
364 places the window's at the location in the new window 

10 at which the drag stopped. Unless the drag is a copy drag, it 
also removes the window from its old desktop. The dragging 
of a window from one desktop to another is illustrated in 
FIGS. 27 and 28 in which the window 180H is shown 
dragged from desktop 182H to a desktop 1821. 

15 If the user clicks on a window in a desktop, step 366 
causes that window to be the selected window in that 
desktop, that is, the window with the current input focus and 
the window that will have that focus the next time the 
desktop is selected. 

20 If the user drags on the title bar of the desktop, step 368 
will move its position relative to the visual display. This 
position will be maintained next time the desktop is dis- 
played again. 

If the user drags on the desktop sizing controls, which 
25 form the frame 345 shown in FIG. 21 around the desktop, 
step 370 will correspondingly change the rectangular dimen- 
sions of the desktop, and these dimensions will be remem- 
bered the next time the desktop is displayed. 

If the user clicks on the desktops close button 342, shown 
30 in FIG. 21, step 372 will remove the desktop from view. 
If the user clicks on the desktop's persistence button 408, 
shown in FIG. 21, step 373 changes the persistence property 
of the desktop to the next persistence property in a cyclical 
sequence of three such properties: non-persistent, semi- 
35 persistent, and persistent, with the current desktop's current 
value of the property being indicated on that button. As 
described above, the persistence property determines under 
what circumstances a desktop is removed from the video 
display. 

40 If the user selects the desktop manager by double-clicking 
on the desktop manager Icon 306, shown in FIG. 18, step 
380 causes step 382 to display the desktop manager window 
312 shown in FIGS. 19 and 20. Then the system waits for 
user input, and in response to a particular input performs a 

45 step, such as step 384, 386, 388, 392, 394, 396, 398, or 400, 
associated with that input. 

If the user places the cursor in the title space after a 
desktop icon 322 and enters textual or editing information, 
step 384 enters or edits the title of the desktop correspond- 

50 ingly. 

If the user selects to make a desktop non-persistent, 
semi-persistent, or persistent by use of the PERSISTENCE 
control 402 in the desktop manager's title bar 404, as shown 
in FIG. 20, then step 386 will change the persistence 

55 property of the desktop accordingly. A persistence icon 406, 
indicates the persistence property of desktops which are 
associated with external locations. It does so with either an 
"N", "S" or "P" shown after the desktop's title in the desktop 
outline, as is shown in FIG. 29. 

60 If the user drags a desktop or window icon to a new 
location in the desktop outline, step 388 causes steps 389 
and 390 to be performed. Step 389 removes the dragged 
desktop or window from its former location in the outline, 
unless the drag is a copy-drag, and places it in the new 

65 location. If the move or copy move inserts an object into a 
portion of the desktop outline other than under the display 
icon 318 of the user making the move, step 390 inserts a 
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"From" entry 432, such as those shown in FIGS. 30, 32, and 
34, labeled with the name of the user making the insertion. 
This "From" entry is placed in the position in outline at 
which the cursor was located at the end of the drag, and the 
actual object being dragged is placed under that "From" 
entry. Step 390 also communicates any change made to a 
portion of the desktop outline not associated with the user 
making the change. It makes this communication over the 
wireless LAN to the computer associated with the part of the 
desktop to which the change has been made. 

If a given outline entry's outline control 410, shown in 
FIG. 29, is a plus sign, indicating that entry has desktops or 
windows indented under it, step 392 will expand the view of 
that given outline entry to show all of the outline entries 
indented immediately under it, if they are not already shown, 
when the user double-clicks on that outline control. If those 
indented entries are already shown, step 392 hides them 
from view. This enables the user to expand or compress 
various portions of the desktop outline at will. 

If the user double-clicks on a given desktop icon 322, step 
394 displays the desktop window as if its associated ID were 
detected. 

If a desktop has a map icon 412, such as those shown in 
FIGS. 29, 30, and 32, and if the user double-clicks on that 
icon, step 396 will show the icon's corresponding map. Such 
a map indicates the physical location of desktops indented 
under that map icon's desktop. 

For example if the user double -clicks on the desktop map 
icon 412 shown in FIG. 30, step 396 shows the desktop map 
window 416 shown in FIG. 31. This map represents all the 
locations in the user's office building associated with user's 
desktops that can be detected by the intra-building locator 
240, shown in FIG. 15. In the preferred embodiment of the 
intra-building location system described above, the wireless 
LAN communicates this map and the information necessary 
to associate locations defined by triangulation between 
transmitters with positions on it. Once communicated, this 
information can be stored in the computer 14A so that the 
map can be rapidly accessed. 

Similarly if the user double-clicks on the desktop map 
icon 412 shown in FIG. 32, step 396 shows map of FIG. 33, 
which illustrates desktop locations defined for the user that 
can be detected by the global positioning locator 242, shown 
in FIG. 15. Such maps for an organization's local area, its 
state, its country, and the world can be stored in a computer 
on the wireless LAN. But a given user's computer only need 
store the maps which he or she frequently uses. 

Desktop maps are provided for intra-building and GPS 
desktop locations. An individual user's intra-building and 
GPS locators 240 and 242 can normally indicate when they 
are at any location in their respective maps. Thus, it is 
possible to associate a desktop with virtually any location, of 
virtually any size, in such a map. 

If the user selects to create a new intra-building or GPS 
location on such a map, step 417 allows the user to draw a 
rectangle 420 representing the size and position of that 
location on the map, and it creates an ID and desktop in 
association with it. These desktops are represented by a 
desktop icon 232 associated with each such location on the 
map. 

If a user select to drag one side 424 of such a location 
rectangle 420, step 418 moves the corresponding rectangle 
and, thus, the location associated with its desktop. 

If the user drags on such a rectangle's sizing drag boxes 
422, step 420 will resize the location's rectangular size to 
correspond to the movement of the side of that rectangle 
which is moved via the drag box. 
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When a user detects bar code IDs with the bar code reader 
170, the computing apparatus 230 can automatically calcu- 
late the approximate location of those bar codes relative to 
an intra-building or GPS map. It can do so from the position 

5 the user was in and the direction his or her head was pointing 
at the time such a bar code pattern was located. A more exact 
location can be calculated if the user views the bar code from 
two or more different angles. If the user so selects, these 
locations can automatically be entered into a intra-building 

10 or GPS desktop map. The locations associated with bar 
codes, however, cannot be changed by dragging or resizing 
an object on the desktop map, since their location can only 
be moved if their associated bar code is moved. In FIG. 31 
the location of such bar codes are indicated by "<" shaped 

15 icons 430. 

If the user double -clicks on any desktop icon in the 
desktop map, step 426 will cause that desktop to be 
displayed, and the user can then drag windows in and out of 
it according to step 364. 

20 Thus, it can be seen that desktop maps make it very easy 
for a user to rapidly associate a desktop with a physical 
location and then to place information in that desktop. When 
the user moves to that location the information placed in its 
desktop will either appear in his or her head mounted 

25 display, or a tone can be selected to sound indicating to the 
user that there is information for the user associated with his 
or her current location. 

. In the example shown in FIG. 31, the user has created 
desktops that become selected whenever he enters certain 

30 rooms, such as the conference room or his boss's office. 
Although he can easily access this information while at any 
location through the portion of the desktop outline shown in 
FIG. 30, the information will be automatically selected 
whenever he enters those rooms. Similarly the user has 

35 associated desktops with certain locations in the GPS map 
shown in FIG. 33. For example, he has associated separate 
desktops with the local supermarket and mall, where he 
places respective shopping lists. He has also placed a GPS 
activated desktop in the driveway of his house. In it he has 

40 placed the window 180J which has a sound alarm associated 
with it, as indicated by the icon 428 shown in FIG. 32. This 
window reminds him of something he is not supposed to 
leave his home without. 

Step 398 of FIG. 17D enables a user to associates a tone 

45 of the type indicated by the icon 428 with a window or 
desktop. This is done by having the user select that window 
or desktop with the cursor and then use the menu bar to 
associate such a tone with it. This tone will sound, if the user 
has his or her sound turned on, whenever the system detects 

50 an ID associated with a desktop which has such a tone 
associated with it or with a window in it. This is useful if the 
user is not wearing his or her head mounted unit when he or 
she enters a location with which important information has 
been associated. The sounding of the alarm will prompt the 

55 user to put on, or turn on, or look at bis or her display. It is 
also useful in embodiments of the invention which do not 
have head mounted visual display, or visual displays at all, 
such as audio -only embodiments of the invention. 

If the user double-clicks on a location icon 326, 32 6 A, or 

60 326 C associated with a given desktop icon, step 400 will use 
the wireless LAN to gather information on, and to display, 
all of the desktops accessible to the user which are associ- 
ated with that location. For example, if the user Bruce 
double-clicks on the location icon 36 6B associated with the 

65 desktop icon 322B, titled "CONFERENCE ROOM" in FIG . 
30, he will see a projection of the desktop outline corre- 
sponding to FIG. 34 showing all the other desktops which 
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might be associated with that same location in portions of When the user places the computer in the position 454 so 

the of the desktop outline associated with other users, such the bar code reader points at the bar code pattern 184A 

as Jim and Fred shown in the example. located above the calendar 186, a program similar to that of 

This ability to see all the desktops which different people FIGS. 17A through 17D displays a desktop associated with 

have associated with a location enables one to see what 5 that bar code pattern. In the example of FIG. 37 this desktop 

desktops windows have been associated with that location, contains a window with a month view in a calendar pro- 

and to place desired windows in the desktop associated with grams. When the user places the computer in the position 

that location by each of one or more users. The user can do 456 so the bar code reader points at the bar code pattern 

this by merely dragging or copy-dragging a window to a 184B placed near the clock 188, the program displays a 

location under a desired user's version of the given desktop, desktop associated with that bar code pattern which, in the 

using steps 388 and step 390, For example, if the user Bruce example, contains a window having a day view of the 

drags the window 180M to under Fred's "Conference" calendar program. Similarly, in the example, when the user 

desktop it will be positioned as is the window 1 80M under points the bar code reader at other bar code patterns shown 

a "From Bruce" entry 432A, as shown in FIG. 34. in FIG. 12, he or she sees desktops containing the windows 

If a user desires to place a window in each desktop described as being associated with those bar code patterns in 

associated with a given location, he or she can do this by 15 the discussion of that figure. 

dragging the window's icon to a position immediately under As is shown at 458, when the user tilts the computer 

the location icon 326B representing that location. If this is forward or backward in the direction 460 the cursor 462 

done that window icon will be inserted under a "From" entry moves correspondingly in the vertical, or Y, direction on his 

at that location, as is indicated by the window 180K shown or her display. When he or she tilts the computer sidewards 

in FIG. 34. When this is done corresponding "From" and 20 in the direction 464 the cursor moves correspondingly in the 

window entry are placed under each desktop associated with horizontal or X, direction on the display. Such titling is 

that location which is accessible to the user making the detected by gravity actuated tilt sensors similar to the sensor 

insertion. 238 shown in FIG. 15. The user can press any of the three 

Preferably the user is allowed to limit who has access to mouse buttons 446, 448, or 450 after having tilted the 

the various portions of his or her desktop outline to prevent 25 computer to move the cursor over a desired screen object so 

others from interfering with selected portions of his or her as to select that object. The user can also keep such a button 

computer work environment. pressed after having so selected an object, to perform a drag 

XIV. Person Specific Messages on Communal Display while moving the cursor by tilting the computer. 

FIGS. 35 and 36 illustrate how the invention can be used The command interpreter for selecting desktops and gen- 

not only to control the display of information on an indi- 30 erating pointing device inputs for such a computer is similar 

vidual's head mounted display, but also on a communal to that described above with regard to FIG. 16, except that 

display 434 which acts as an electronic bulletin board. In this it uses mouse button input instead of eye blinks to generate 

case the communal display is treated in the desktop outline mouse button down signals. When a separate button 466 is 

just as is each user's individual display. For example, the pressed, motion of the computer is not used to move the 

display "Front Door Msg. Screen" shown in FIG. 29 is a 35 cursor, allowing extended cursor motion in any given direc- 

communal display. As is indicated in FIG. 36, in which that lion without requiring a corresponding net motion in that 

display's part of the desktop outline has been expanded, it direction, the same way that the closing of both eyes does 

has the same type of outline structure as does the individual with the program of FIG. 16. 

display associated with "Bruce". The embodiment of the invention shown in FIG. 37 

Since such communal displays are normally stationary, 40 provides a hand held computer in which a user can rapidly 

the desktops they display are normally associated with bar select windows and perform functions upon them without 

coded patterns worn on individual identity badges, such as requiring a separate pointing device. If the computer 

the bar code 184F shown on the person 204 in FIGS. 12 and includes speech recognition, the user would be able to 

35. When no such bar code is detected within the view of the perform all computer functions with just the one hand 

bar code reader 170A located with the display, the display 45 required to hold that computer. 

shows the windows, if any, associated with the background XVI. Mdeo Recognition of Pointing Toward External Loca- 

desktop icon 360 shown in FIG. 36. When the system detects tions or Objects 

a person wearing a bar code pattern 184F, step 245A, 304, FIG. 38 shows another embodiment of the invention in 
330, and 338 of FIG. 17A will display the desktop, if any, which a user 468 can select desktops merely by pointing his 
associated with that user on the display 434. 50 or her hand 469 at their associated external locations. This 
In other embodiments of the invention, the bar code embodiment's computing apparatus 470 includes a corn- 
reader 170A can be replaced with a video camera and visual puter (not shown in the figure), such as a SPARCstation 
recognition system, similar to that shown in FIG. 14, which computer of the type described above by Sun 
recognizes faces or other identifying visual features of Microcomputer, Inc., a display screen 472 and one or more 
individual people and uses them to determine which desk- 55 video cameras 474 connected to it. 
tops to 'display. FIG. 39 show a high level flow chart of a part of the 

XV. Hand Held Display with Motion Controlled Cursor and program loop 250A which is used in the embodiment of the 
Window Selection invention shown in FIG. 38. The loop 250A is a modified 

FIG. 37 illustrates how the present invention can be version of the same command loop 250 which is partially 

beneficially used in a hand held computer 440 having a 60 shown in FIGS. 16 and 17A-17D. 

visual display 442. In this embodiment of the invention the Step 482 processes the video image or images from the 

computer has a microphone 166 and speaker 168 hidden apparatus's one or more video cameras, 

behind openings 444, left, middle, and right mouse button, Step 484 scans those one or more images for patterns that 

450, 448, and 446, respectively, and a bar code reader 170B. match arm-hand-finger models stored in the computer's 

It also contains a tilt detector like the detector 238 shown in 65 memory. These models includes separate component models 

FIG. 15, except that it detects side-to-side tilting as well as for arms, hands, and fingers shown in different angles and 

front-to-back tilting. models for allowed linkages between such body parts. 
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Step 486 tests to see if the best scoring arm-hand -finger 
model corresponds to a pointing gesture, and if so it causes 
steps 488 and 490 to be performed. In the embodiment 
shown, a pointing gesture occurs when the user sticks his or 
her index finger out strait, with the thumb sticking up and the 5 
other fingers folded, as is indicated by the hand 469 shown 
in FIG. 38. 

If the best scoring arm-hand-finger model corresponds to 
such a pointing gesture, step 488 estimates the location of 
the finger of the best scoring arm-hand-finger model and the 10 
direction in which it is pointing. If there are two cameras, 
location of the finger can be calculated by triangulation. If 
not, it can be estimated by size of the finger's image relative 
to the user's finger, if the system has been trained to know 
the user's finger size, or to the size of an average finger, if 15 
not. The direction of pointing can be calculated from the 
angle in which the finger of the best scoring arm-hand-finger 
model appears pointed in the image or images captured by 
the one or more camera's 474. 

Once step 488 is complete, step 492 calculates a limited 20 
angular pointing zone which extends out radially from the 
calculated location of the finger in the direction in which that 
finger is calculated to point. 

If the program is in a mode in which the user is training 
the system to. know where a location associated with a 25 
desktop is, step 492 records the time when the user is first 
detected pointing in that direction under such circumstances. 
On successive passes through the loop, the step counts the 
length of time the user has continued pointing in that 
direction during the training mode. If, on any given pass 30 
through the loop, step 492 indicates the pointing gesture has 
been pointing in the same direction for over a quarter of a 
second, step 493 causes the desktop location being trained to 
be defined as the volume of the intersection of the pointing 
zone corresponding to the current pointing direction with 35 
any previous pointing zones for the given desktop location. 

During this training process the user is instructed to point 
at the location to be associated with the desktop from at least 
three different directions, including from different vertical 
and horizontal positions. The user is also instructed to hold 40 
his or her hand in each such pointing gesture for at lest a 
quarter second. The intersection of the resulting different 
pointing zones created in different passes through the loop 
480 will define a location for the desktop which has a limited 
extent in all three spatial dimensions. 45 

If the program is in the command recognition mode, the 
mode in which it responds to user gestures to control desktop 
display and cursor functions, the test of step 494 will be met, 
causing steps 496 and 500 to be performed. 

On each pass through the loop 250A in which the system 50 
is in command recognition mode step 496 determines if 1) 
the user has pointed at a given location on the screen for at 
least a quarter of a second and 2) has not pointed his or her 
finger at a location which is substantially off the screen since 
the end of that quarter of a second. If the condition of this 55 
test are currently met, either step 497 or 498 can be 
performed. 

If the best scoring arm-hand-finger model is a pointing 
gesture, step 497 moves the display's mouse cursor to 
correspond to the location at which the center of the finger 60 
pointing zone intersects the screen. Preferably a smoothing 
algorithm, of the type described above with regard to the 
graphics processor 54, is used to prevent normal unsteadi- 
ness of the hand from undesirably shaking the mouse cursor. 

If the best scoring arm-hand-finger model corresponds to 65 
a mouse button gesture, step 498 generates the correspond- 
ing mouse button input. Pointing the thumb down so it 
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points in substantially the same direction as the pointing 
index finger is interpreted as a left mouse button down. 
Pointing the thumb down toward the palm at approximately 
a ninety degree angle to the index finger is interpreted as a 
right mouse button down. And pointing all the fingers out 
parallel to the index finger is interpreted as the middle mouse 
button down. When such gestures are made the mouse 
should remain stationary unless the index finger moves 
excessively, so as to compensate for the shaking of the index 
finger which will normally result during such movement of 
other fingers. 

If during command recognition mode a pointing gesture 
has defined substantially the same pointing zone for more 
than a quarter of a second and that pointing zone does not 
intersect the screen 472 or an area immediately around it, 
step 500 enables step 502 or 506 to be performed. 

If the test of step 500 determines that the pointing zone 
detected for over a quarter of a second intersects a location 
associated with a desktop, step 502 selects and shows the 
desktop on the display 472 in a manner similar to that of step 
304 of FIG. 17A. 

If the pointing zone does not intersect any location 
associated with a desktop, and if the pointing gesture defin- 
ing that zone is continued for a total of two seconds, step 504 
pops up the training mode dialog box which gives the user 
the option of defining a new location and the desktop 
associated with it. If the user selects to enter this training 
mode, steps 492 and 493, described above, will be per- 
formed. 

The program of FIG. 39 also includes steps, not shown in 
that figure, similar to those of FIGS. 17 A through 17D for 
associating windows with desktops and for displaying, 
navigating, and editing a desktop outline. 

It can be seen that the apparatus and method of FIGS. 38 
and 39 make it possible for a user with a traditional display 
to make use of the present invention's ability to associate 
computational objects with locations and then to select those 
objects merely by pointing at 1 them. If the computer 14B 
includes speech recognition, this system would enable a user 
to perform all the operations necessary to operate a com- 
puter from a distance without ever needing to touch any part 
of the computer apparatus. 
XVII. Alternate Embodiments 

The present invention provides novel ways to navigate 
through, rapidly select, and easily interact which computer 
generated information and objects. It should be understood 
that the novel aspects of the invention can by realized in 
many different ways in many different embodiments. 

For example, in other embodiments the view spaces 
shown in FIGS. 2, 6, 7, and 12 can be made three dimen- 
sional and the user can navigate in them by three dimen- 
sional motion. Such motion could be sensed by any appro- 
priate means, including the ultrasonic sensing system 
described with regard to FIGS. 1-4, a 3-D radio triangula- 
tion system, such as the intra-building system described 
above, by having the bar code reader 170 detect the fre- 
quency with which light is modulated by the reflection off a 
bar code pattern to determine changes in distance from such 
a pattern, and by having video cameras such as those shown 
in FIG. 39 triangulate or calculate changes in distance from 
objects by changes in the size of the images of those objects. 

In some embodiments of the invention characteristics are 
combined from the different types of view spaces described 
above, the continuous view space of FIG. 2, the quantized 
path-independent view space of FIG. 6, the quantized hyper 
view space of FIG. 7, and the quantized, location-dependent, 
view space shown in FIG. 12. For example, a screen shown 
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in any one of these different types of view spaces could The dimensions and size of the virtual view space 40, 

become an entrance to another one of those types of view frame buffer 42 and view window 36 are all arbitrary. In 

spaces, in a manner similar to that described above with some embodiments of the invention the screen buffer might 

regard to screen 114B of FIG. 6. be no larger than the screen of the video display. In some 

View spaces of the type shown in FIGS. 2, 6, 7, 12 can be 5 embodiments of the invention the view space stores, and the 

viewed with any type of display including: head mounted v is Ual display shows, monochrome images. In others they 

displays which shut out the surrounding physical world, as store and sho w color images. The graphics processor 54 can 

shown in FIG. 2; head mounted displays .which i allow the be modified to perform a nurnber of standard graphics 

user to see the surrounding word, as shown in i FIGS. 9 and relatcd functio such as shading , anti-aliasing, 

10; small portable displays, as shown in FIG. 37; desk « . f - , 

« j. . c ■ in ii .j 10 illumination, etc. 

mounted displays as shown in FIG. 39: or wall mounted , , . . , . , . c 

displays. Some futurists are talldng about building light . 11 f oul * be appreciated that the embodiments of the 

valves or light emitting elements into contact lenses Whin, mvcnt J on dcscri ^ cd abo u vc allows * uscr J° a succcs " 

and if, such contact lens video display arc built, their use for Sl ° n of quantized views by moving through successive zones 

the purpose described above would come within the scope of of motlOD wlthout «q™ng a separate selectmg step m each 

this invention. 15 sucn zonc - This * s truc regard to the navigation of the 

The measuring step 92 of FIG. 5, the position change s P aces stl0Wn in FIGS - 6 and 7 WDen performed according 

calculating steps of FIG. 8, and the detecting steps 158 and to the flow chart of FIG. 8, as well as the navigation of the 

254A of FIG. 11 and of FIGS. 17A-17B, respectively, can space shown in FIG. 12 when performed according to the 

all be performed in ways other than by measuring motion or flow charts of FIGS. 16 and 17A-17D. This is advantageous 

position of the user's head. This includes detecting the 20 because it lets the user scan visual information at a very high 

position of a portable computer or visual display as shown rate, without slowing him or her down by requiring an extra 

in FIG. 37, detecting the position of the user's portable selection step to see each new piece of the visual space. In 

computer, such as by the intra-building and global position- some embodiments of the invention, however, the user 

ing locators discussed with regard to FIGS. 15 and 30-33, or might be required to click a mouse button or perform some 

detecting the position of a part of the user's body, as shown 25 other selection step to see each new piece of the visual space, 

in FIGS. 38 and 39. for example, to prevent his or her view from being changed 

As is indicated in FIG. 35, these position detecting steps unless specifically desired, 

can also be used to alter the computers output by detecting It should be understood that the information navigational 

changes in the relative position between the user, or a sensor, methods described above will often be combined with other 

and a given location or object. Detection of the position or 30 navigational methods, such as associating certain windows 

motion of the user, parts of the user's body, the user's or desktops with keys of a computer keyboard, 

computer, a sensor, or an external object can all be detected In alternate embodiments of the aspect of the invention 

by virtually any means and still come within the scope of the shown with regard to FIG. 11 and the figures that follow it, 

present invention. items other than desktops are associated with external loca- 

The external locations with which communication links 35 tions. For example, in some such embodiments windows are 

are associated according to the aspect of the invention associated directly with external locations, rather than being 

shown in FIG. 11 and many of the figures that follow include placed at desired locations in desktops associated with such 

locations associated with objects which can be detected by locations. In other embodiments of this aspect of the inven- 

a sensor such as the bar code reader, infrared detector, or tion computational objects other than GUI windows are 

video camera of FIGS. 12, 13, and 14, respectively. Other 40 associated with external locations, including character based 

types of objects and sensor could be used, such as visible windows, images, text messages, audio messages, OOP type 

lights and photo detectors, ultrasonic generators and objects, changes in input or output direction, different loca- 

detectors, the human voice and a speaker identification tions within a document, changes in modes of user interface 

module, etc. The location can be defined relative to a spatial operation, the turning of a display on or off, the zooming of 

location system such as the intra-building and GPS systems 45 a display's field of vision in or out, inputs in a command 

described above. It also can be defined relative to a user's grammar, etc. 

own body. For example, pointing the user's head to his or Alternate embodiment of the invention can operate with 

her left shoulder could select one communication link and different types of hardware than that described above. As has 

pointing it to his or her right shoulder could select another. been mentioned, different types of visual displays and 

The computation which equates changes in motion or 50 motion or position sensing apparatus can be used. Comput- 

position with changes in display can be varied in different ers other than those manufactured by Sun Microcomputer, 

embodiments of the invention. For example, the amount of Inc. described above can be used. For example, pocket sized 

head or other user movement required to navigate from one or wrist mounted computers could be used. The computers 

screen to the next in the view spaces of FIGS. 2, 6, and 7 can could be multiprocessing computers or non- Von Neumann 

be varied greatly. In some situations it is advantageous to 55 computers. Neural net or fuzzy logic computer could be 

make a small amount of user motion cause the transition used, for example, to automatically associate a communi- 

from one screen to the next. In others it is desirable to allow cation link with a given location or object if the user often 

the user a wide degree of movement without changing uses that link when pointing at or near that location or object, 

screens. For example, when very small amounts of motion In different embodiments of the invention different sub- 

are sufficient to cause screen changes, movement through a 60 sets or superset of the input and output devices shown in 

quantized space of the type shown in FIGS. 6 or 7 can be FIG. 15 can be used. For example, in some embodiments of 

used to create a continuous "motion picture" image. the invention an individual user's system is not networked, 

The access step 98 of FIG. 5 is implemented by the scan as described above with regard to FIG. 15, and, thus, has no 

line generator 60 under the supervision of the graphics means for placing windows in, or seeing what windows are 

processor 54. It should be understood that in other embodi- 65 located in, desktops associated with other users. In other 

ments of the invention different methods for generating scan embodiments, the user only carries a display or a smart 

lines can be used. terminal and much of the computation which drives his or 
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her display is performed in a stationery computer, either one 
associated with the individual user alone, or a time sharing 
machine. The connection to such a stationary computer can 
be either by wire, optical, or radio link. 

It should be understood that in other embodiments of the 5 
invention other types of user interface can be used. For 
example, the various continuous, quantized, and location 
dependent view spaces described above could be used in a 
character based user interface. 

The command interpreter shown in FIG. 16 used to 
control a head-mounted, see-through display could use input 
other than eye closings to control its operations. For 
example, in other embodiments eye ball direction, speech, 
head nods, or other body gestures could be used. Similarly 
the command gestures used with embodiments of the inven- 
tion like that of FIGS. 38 and 39 which views the motion of 15 
body parts can differ. For example, in a system monitoring 
head position with a video camera, blinks or facial expres- 
sions can be used to simulate mouse clicks or indicate when 
pointing is to be interpreted as a command. 

It should be understood that the desktops described above 20 
can be used for other purposes besides associating windows 
with locations. They also provide a convenient way to 
organize computer work environments. The display of such 
desktops can be triggered by conditions other than the 
detection of locations. For example, the desktops under the 2 5 
desktop icon 322C shown in FIG. 29 are timed desktops that 
pop up at pre -specified times on specified days or on 
reoccurring patterns of days. Such other conditions can be 
combined with the detection of a location to trigger the 
display of a desktop. For example, a given desktop might 30 
only be popped up when a given location is detected if it is 
also a given day, or if a voice associated with a given person 
is also recognized, or if given words have also been recog- 
nized. 

In other embodiments of the invention different methods 35 
besides the use of a desktop outline could be used to keep 
track of what communication links had been associated with 
what locations and to make changes in those associations. 
For example, where the number of locations is relatively 
small there is no need for such an outline at all. 40 

It should be noted that the devices described in reference 
to FIGS. 1-39 are only exemplary. It is within the scope of 
the present invention to use any device to implement the 
above-mentioned functions. Thus, while the invention has 
been described in relationship to the embodiment shown in 45 
the accompanying figures, other embodiments, alternatives 
and modifications will be apparent to those skilled in the art. 

What is claimed is: 

1. A head mounted display apparatus comprising: 

a display for receiving a signal representing a visual 50 
image and for projecting a corresponding visual image 
which can be seen by a user wearing said apparatus; 

a directional detector for generating an object detection 
signal when said detector receives a certain signals 
from a physical object in response to said detector 55 
being pointed at said object, said detector being 
mounted on said apparatus so a user who is wearing the 
apparatus can point the detector by moving his or her 
head. 

2. A head mounted display apparatus as in claim 1, further 60 
comprising a see through window to enable a user to see his 

or her physical surroundings when wearing the apparatus. 

3. Ahead mounted display apparatus as in claim 1, further 
comprising an additional input device to enable a user 
wearing the apparatus to selectively generate computer 65 
readable signals in addition to those generated by said 
detector. 
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4. A head mounted display apparatus as in claim 1 
wherein said detector points in the same general direction as 
the face of a user wearing said apparatus. 

5. A head mounted display as in claim 1, wherein: 

the physical objects radiate electromagnetic radiation at a 
sufficient frequency that it travels in a substantially 
straight line; 

-the directional detector has a limited angular view from 

which it can receive radiation; 
the user can point the angular view by moving his or her 
head; and 

the directional detector generates the object detection 
signal when the detector is pointed at one of said 
physical object so that it can receive radiation coming 
straight from such an objects. 

6. A method of providing a computer graphical user 
interface comprising the steps of: 

computing a view space in which graphical user interface 
windows each generated by a separate computer pro- 
cess can be selectively located by a user, said view 
space having a plurality of discrete portions each of 
which is large enough lo hold a plurality of said 
windows generated by separate computer processes; 

computing a plurality of said graphical user interface 
windows; 

enabling a user to selectively locate a variable number of 
said windows in each of said view space portions and 
to selectively move said windows between view space 
portions by use of a point and click interface; 

displaying a subset of said view space portions to the user 
with a head-mounted visual display; 

sensing motion of a user's head; 

changing the portions of the view space displayed to the 
user, in discrete units of view space portions, in 
response to changes sensed in the position of the user's 
head. 

7. The method of claim 6 wherein said step of enabling a 
user to selectively locate and move windows includes 
enabling a user to drag a window with a pointing device to 
a desired portion in said view space. 

8. The method of claim 6 wherein: 

said step of sensing motion of a user's head includes 
sensing rotational movement of a user's head; and 

said step of changing the portions of the view space which 
are displayed includes making such changes in 
response to the detection of said rotational motion, so 
a user can change the portion of the view space shown 
without moving the majority of his or her body, merely 
by rotating his or her head. 

9. The method of claim 6, further comprising the step of 
detecting eye movements of the user and using the detec- 
tions of said eye movements to enable the user to interact 
with windows in the portion of the workspace displayed to 
him or her. 

10. The method of claim 6 wherein: 

the sensing of motion of a user's head, including dividing 
head motions into at least a first and second class of 
head motion; 

the method makes said changes in the portions of the view 
space displayed to the user in response to head motions 
of said first class but not head motions of said second 
class, so as to enable a user to move further in the view 
space in a given direction than would otherwise be 
possible by causing successive head motions in the 
given direction to be of the first class which are counted 
toward such view space motion, and causing one or 
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more intervening head motions in the opposite 
direction, made so as to prevent the head from being 
over extended in the given direction, to be of the second 
class, so as to go uncounted toward such view space 
motion. 5 

11. A method of providing a computer graphical user 
interface comprising the steps of: 

computing a computer generated visual space; 

displaying a subset of said visual space to the user; 

sensing user motion including monitoring motion of a 10 
user's head, said sensing including dividing head 
motions into at least a first and second class of head 
motion; 

allowing a user to navigate through said visual space by 
changing the subset of the visual space displayed in 
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response to head motions of said first class, but not in 
response to head motions of said second class, so as to 
enable a user to move further in the visual space in a 
given direction than would otherwise he possible hy 
causing successive head motions in the given direction 
to be of the first class which are counted toward such 
visual space motion, and causing one or more inter- 
vening head motions in the opposite direction, made so 
as to prevent the head from being over extended in the 
given direction, to be of the second class, so as to go 
uncounted toward such visual space motion, 
12. A method as in claim 11 wherein said step of display- 
ing uses a head mounted display. 

***** 
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